Image generation method and apparatus based on dynamic filter bank, device, and medium

Abstract

The application relates to the technical field of image processing, and discloses an image generation method and device based on a dynamic filter set, equipment and a medium, which comprises the following steps: obtaining generation input information and condition input information, forming an initial feature map and a condition latent representation; forming a to-be-up-sampled feature map based on the initial feature map, extracting a global feature statistic quantity, combining the global feature statistic quantity with the condition latent representation to form a filter generation feature representation; generating a dynamic filter set and attention modulation information, forming a modulated dynamic filter set; using the modulated dynamic filter set to form an up-sampled feature map and converting the up-sampled feature map into a generated image. The application can be applied to business scenarios such as financial technology and medical health, a dynamic filter set is formed through a condition latent representation and a global feature statistic quantity, the dynamic filter set is modulated by using attention modulation information, the up-sampling process is combined with input condition semantics and current feature content, and therefore the situation that a generated image local detail is inconsistent with condition semantics is reduced.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image generation method, apparatus, device, and medium based on dynamic filter banks. Background Technology

[0002] Conditional image generation techniques typically generate images corresponding to conditional content based on category labels, text descriptions, or other conditional information. Existing generative networks often employ fixed interpolation, transposed convolution, or fixed subpixel convolution to enhance the resolution of feature maps. In these approaches, the filtering parameters used during the upsampling stage remain fixed after model training, and the same feature expansion method is applied regardless of the conditional content. This makes it difficult for conditional information to consistently contribute to the formation of high-resolution details. Especially in scenarios requiring the generation of local textures, boundary structures, and object compositions based on fine-grained semantics, the generated images are prone to inconsistencies between local details and the input conditions.

[0003] In the fintech business, financial institutions can generate images for financial products, promotional visuals, and digital service interfaces based on insurance product types, wealth management themes, customer service scenario tags, marketing copy descriptions, or investment advisory presentation needs. Existing image generation methods typically rely on fixed upsampling to create high-resolution content when generating product identification areas, coverage display areas, asset allocation themes, and customer interaction interface details. When the input conditions change from an insurance coverage scenario to a wealth management scenario, or from a conservative product display requirement to an equity product display requirement, the fixed upsampling method struggles to adapt the formation process of local image details according to the semantic conditions, easily leading to deviations between the business elements and visual focus in the generated images and the input financial scenario description.

[0004] In the healthcare field, conditional image generation can be used to create medical imaging-assisted images, lesion display images, and health education visual content based on image modality information, lesion category labels, examination description text, or tissue structure prompts. Existing image generation methods, when using fixed upsampling to restore or synthesize tissue texture, organ contours, and abnormal region boundaries, struggle to fully reflect the impact of different lesion conditions and image descriptions on high-resolution local features during the upsampling stage. This leads to issues such as blurred lesion edges and inconsistencies between tissue details and conditional descriptions in the generated images, affecting the accuracy of the conditional image generation results in representing input healthcare information. Summary of the Invention

[0005] The main objective of this invention is to provide an image generation method, apparatus, device, and storage medium based on a dynamic filter bank, aiming to solve the technical problem that existing conditional image generation techniques use fixed filtering parameters for upsampling, making it difficult to change the formation process of high-resolution local features according to input conditions, resulting in inconsistencies between the generated image details and the semantics of the input conditions.

[0006] To achieve the above objectives, the present invention provides an image generation method based on a dynamic filter bank, comprising: Obtain the generated input information and the conditional input information, convert the generated input information into an initial feature map, and encode the conditional input information into a conditional latent representation; The initial feature map is transformed to generate a feature map to be upsampled. Global feature statistics are extracted from the feature map to be upsampled. The conditional latent representation and the global feature statistics are combined into a filter to generate a feature representation. A dynamic filter bank is generated based on the filter generation feature representation; The upsampled feature map is downsampled to generate a downsampled feature map, and attention modulation information is generated based on the downsampled feature map and the conditional latent representation. Based on the attention modulation information, the dynamic filter bank is modulated to generate a modulated dynamic filter bank. Based on the modulated dynamic filter bank, the feature map to be upsampled is upsampled to generate an upsampled feature map. The upsampled feature map is converted into a generated image.

[0007] Furthermore, to achieve the above objectives, the present invention provides an image generation apparatus based on a dynamic filter bank, comprising: The input representation module is used to acquire generated input information and conditional input information, convert the generated input information into an initial feature map, and encode the conditional input information into a conditional latent representation; The filtering feature module is used to perform feature transformation on the initial feature map to generate a feature map to be upsampled, extract global feature statistics from the feature map to be upsampled, and combine the conditional latent representation and the global feature statistics into a filter to generate a feature representation. A filter generation module is used to generate a dynamic filter bank based on the filter generation feature representation. Attention modulation module is used to downsample the feature map to be upsampled to generate a downsampled feature map, and generate attention modulation information based on the downsampled feature map and the conditional latent representation; The modulation upsampling module is used to modulate the dynamic filter bank based on the attention modulation information to generate a modulated dynamic filter bank, and to upsample the feature map to be upsampled based on the modulated dynamic filter bank to generate an upsampled feature map. An image output module is used to convert the upsampled feature map into a generated image.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and an image generation program based on a dynamic filter bank stored in the memory and executable on the processor, wherein when the image generation program based on a dynamic filter bank is executed by the processor, it implements the steps of the image generation method based on a dynamic filter bank as described above.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing an image generation program based on a dynamic filter bank, wherein the image generation program based on a dynamic filter bank, when executed by a processor, implements the steps of the image generation method based on a dynamic filter bank as described above.

[0010] Beneficial Effects: This invention relates to the field of image processing technology and discloses an image generation method, apparatus, device, and medium based on a dynamic filter bank. The method includes: acquiring generation input information and conditional input information to form an initial feature map and a conditional latent representation; forming a feature map to be upsampled based on the initial feature map, extracting global feature statistics and combining them with the conditional latent representation to form a filter-generated feature representation; generating a dynamic filter bank and attention modulation information to form a modulated dynamic filter bank; using the modulated dynamic filter bank to form an upsampled feature map and converting it into a generated image. This invention can be applied to business scenarios such as fintech and healthcare. By forming a dynamic filter bank through conditional latent representation and global feature statistics, and modulating the dynamic filter bank using attention modulation information, the upsampling process combines the input conditional semantics and the current feature content, thereby reducing inconsistencies between local details and conditional semantics in the generated image. Attached Figure Description

[0011] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for an image generation method based on a dynamic filter bank according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the image generation method based on a dynamic filter bank according to the present invention; Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the image generation device based on a dynamic filter bank of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0012] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0013] The image generation method based on dynamic filter banks provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain the generation input information and conditional input information from the client to form an initial feature map and a conditional latent representation; based on the initial feature map, it forms a feature map to be upsampled, extracts global feature statistics and combines them with the conditional latent representation to form a filter-generated feature representation; it generates a dynamic filter bank and attention modulation information to form a modulated dynamic filter bank; it uses the modulated dynamic filter bank to form an upsampled feature map and converts it into a generated image. This invention can be applied to business scenarios such as fintech and healthcare. By forming a dynamic filter bank through conditional latent representation and global feature statistics, and modulating the dynamic filter bank using attention modulation information, the upsampling process combines the input conditional semantics and the current feature content, thereby reducing the inconsistency between local details and conditional semantics in the generated image. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0014] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the image generation method based on a dynamic filter bank provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0015] like Figure 2 As shown, the image generation method based on dynamic filter banks proposed in this invention includes the following steps: S10, acquire the generated input information and the conditional input information, convert the generated input information into an initial feature map, and encode the conditional input information into a conditional latent representation; In this embodiment, the generated input information carries the content data of the image to be formed, which can be a feature vector, low-resolution image features, or features of the region to be completed. During acquisition, the generated input information can be read from the processing request, image storage area, or pre-processing results, and configured into a data structure usable for feature mapping according to the data dimensions and number of channels.

[0016] The initial feature map is used to organize the feature values ​​in the generated input information according to spatial location and feature channels. In specific implementation, the input mapping unit can be used to transform the dimensions of the generated input information to form channelized feature data, and then the channelized feature data can be arranged according to the spatial row and column relationship to form the initial feature map. For generated input information with a given spatial structure, the feature channels can be adjusted by convolutional mapping while preserving the spatial arrangement relationship to form the initial feature map.

[0017] The conditional input information carries the conditional meaning corresponding to the image content, and can be text data, category labels, attribute fields, or a data set formed by a combination of multiple conditional contents. During acquisition, text content or identifier content can be read from the conditional fields in the processing request and converted into data units suitable for encoding.

[0018] Conditional latent representations are used to carry the category meaning, descriptive meaning, and attribute combination relationships in conditional input information in the form of continuous features. In specific implementation, the conditional input information can be split or converted into conditional data units. The corresponding embedding vectors are read according to each conditional data unit, and the embedding vectors are input into the encoding unit for semantic fusion. The encoding results are then aggregated to form the conditional latent representation.

[0019] In the fintech business, processing requests can carry page visual feature data used to generate insurance product display images, along with insurance product type, coverage theme text, and customer service page theme fields. The page visual feature data serves as the input information, undergoing dimensional transformation and spatial arrangement to form an initial feature map. The insurance product type, coverage theme text, and customer service page theme fields serve as conditional input information, undergoing embedding, reading, and fusion encoding to form conditional latent representations, ensuring that the content expression of the insurance display and the semantics of insurance coverage respectively form corresponding data results.

[0020] In the healthcare business domain, processing requests can carry low-resolution tissue image features for generating image-assisted visualizations, along with image modality markers, tissue region hints, and descriptive text for abnormal regions. The low-resolution tissue image features serve as input information, undergoing convolutional mapping to form an initial feature map. The image modality markers, tissue region hints, and descriptive text for abnormal regions serve as conditional input information, undergoing embedding, reading, encoding, and aggregation to form a conditional latent representation, thus ensuring that the regional content representation of the tissue image and the medical descriptive semantics correspond to the respective data results.

[0021] This embodiment converts the generated input information into an initial feature map with spatial location and feature channel relationships, allowing the content data of the image to be formed to be stored in a regionally distributed form. By encoding the conditional input information into a conditional latent representation, the conditional meaning in text, labels, or attribute fields can be transformed into a continuous semantic expression. Thus, the image content expression and conditional semantic expression each form a clear data result, reducing the possibility of the conditional meaning being weakened during the image content formation process, which could lead to content deviation.

[0022] S20, perform feature transformation on the initial feature map to generate a feature map to be upsampled, extract global feature statistics from the feature map to be upsampled, and combine the conditional latent representation and the global feature statistics into a filter to generate feature representation; In this embodiment, the initial feature map adopts a feature data format jointly organized by spatial location and feature channels. Spatial location records the arrangement relationship between image regions, while feature channels record the contour, texture, color, or structural response corresponding to the same region. Feature transformation can be accomplished through convolutional mapping, residual mapping, channel blending, or multi-layer feature mapping, adjusting the feature response of each spatial location while maintaining the spatial arrangement relationship to form the feature map to be upsampled. The feature map to be upsampled retains the transformed spatial content distribution and serves as the image content representation in the current processing.

[0023] Global feature statistics are generated from the feature map to be upsampled and represent the overall distribution of the current image content across each feature channel. In practice, for each feature channel, the feature values ​​corresponding to each spatial location can be averaged and aggregated to form channel statistical feature values. These channel statistical feature values ​​are then organized according to the order of the feature channels to form the global feature statistics. Alternatively, weighted aggregation can be performed based on the response intensity at each spatial location, allowing the global feature statistics to retain channel distribution information corresponding to areas with more pronounced responses.

[0024] Conditional latent representations, using continuous feature forms to carry the semantic meaning of the conditional content, belong to the already formed conditional expression data. Combining conditional latent representations with global feature statistics can be done by concatenating along the feature dimensions, or by mapping them to their corresponding dimensions and then fusing them to form a filter-generated feature representation. This filter-generated feature representation simultaneously includes the semantic expression of the conditional content and the overall content distribution of the feature map to be upsampled, reducing the inadequacy of reflecting conditional differences when data results are formed solely based on image content.

[0025] Regional features of the surface. After feature transformation of the initial feature map, a feature map to be upsampled is formed, which includes the product display area, the protection description area, and the visual theme area response. The spatial feature values ​​in each feature channel are aggregated to form a global feature statistic. The conditional latent representation formed by the insurance product type, protection theme, or wealth management display theme is combined with the global feature statistic to form a filter-generated feature representation that simultaneously includes financial business semantics and the distribution of screen content.

[0026] In the healthcare field, the initial feature map can carry regional features of tissue image display content or health education images. After feature transformation of the initial feature map, a feature map to be upsampled is formed, which includes the responses of tissue regions, structural boundaries, and abnormal indication regions. The channel responses in the feature map to be upsampled are aggregated to form a global feature statistic. The conditional latent representation formed by image modality, tissue region indication, or abnormal description is combined with the global feature statistic to form a filter-generated feature representation that simultaneously incorporates healthcare semantics and image content distribution.

[0027] This embodiment transforms the initial feature map so that the upsampled feature map can retain the adjusted spatial content distribution. By forming global feature statistics from the upsampled feature map, the overall response of the current image content in each feature channel is centrally expressed. By combining the conditional latent representation with the global feature statistics to form a filter to generate feature representation, the resulting data contains both conditional semantics and image content distribution, thereby reducing the detail deviation caused by the separation of conditional semantics and image content expression.

[0028] S30, Generate a dynamic filter bank based on the filter generation feature representation; In this embodiment, the filter-generated feature representation carries two types of information: semantic features corresponding to the conditional content and overall statistical features corresponding to the image content. The dynamic filter bank is used to form a set of filter parameters that can change with the input content. This set of filter parameters includes spatial arrangement relationships, input channel correspondences, and output channel correspondences, enabling different input content to correspond to different filter responses.

[0029] When generating a dynamic filter bank, a filter generation network can be set up. This network receives the feature representations generated by the filters and transforms the feature data into filter parameter data through multi-layer parameter mapping. Parameter mapping can employ a combination of linear mapping, nonlinear activation, and dimensionality expansion to project semantic and content differences in the filter-generated feature representations into the filter parameter space. To ensure the output has a clear filtering structure, filter space arrangement information, input channel arrangement information, and output channel arrangement information also need to be configured. These arrangement information are used to organize the parameter mapping results into a dynamic filter bank. Each set of filter parameters in the dynamic filter bank corresponds to a type of spatial position response and channel response, thus reflecting the filtering differences caused by changes in input content. To maintain a stable data distribution of the filter parameters formed under different input conditions, the generated filter parameters can be normalized, scaled, and offset. Normalization is used to unify the parameter amplitude range, scaling is used to adjust the parameter response intensity, and offset is used to adjust the parameter baseline value. The resulting dynamic filter bank is more suitable for subsequent feature transformations.

[0030] In one implementation, the filter generation network employs a multi-layer mapping structure. The filter generation feature representation is input to the first mapping layer to form intermediate parameter features. These intermediate parameter features are then passed through subsequent mapping layers for further transformation, ultimately outputting filter parameter data. Based on pre-configured filter space dimensions, the number of input channels, and the number of output channels, the filter parameter data is reorganized to form a dynamic filter bank. Subsequently, all filter parameters in the dynamic filter bank are normalized in magnitude and corrected using scaling and offset parameters, resulting in a dynamic filter bank ready for subsequent use.

[0031] In another implementation, the filter generation network employs a branch-mapping structure. The filter generation feature representation first passes through a shared mapping layer to form common parameter features, then is fed into spatial parameter branches, input channel parameter branches, and output channel parameter branches respectively, forming parameter data of different dimensions. Finally, these are concatenated according to spatial and channel dimensions to form a dynamic filter bank. For scenarios with significant differences in input content, normalization and channel scale adjustment can be added after concatenation to ensure consistency in parameter distribution within the dynamic filter bank.

[0032] This embodiment transforms the filter-generated feature representation into a dynamic filter bank through a filter generation network, allowing semantic and content features to jointly participate in the filter parameter formation process. Combined with parameter recombination, normalization, scaling, and offsetting, the dynamic filter bank can form a parameter expression corresponding to the input content, thereby enhancing the responsiveness of subsequent filtering processes to changes in input conditions.

[0033] S40, downsample the feature map to be upsampled to generate a downsampled feature map, and generate attention modulation information based on the downsampled feature map and the conditional latent representation; In this embodiment, the feature map to be upsampled has a spatial location dimension and a feature channel dimension. The spatial location carries the content response of the local region, while the feature channels carry local texture, boundary, and structural information. Downsampling can be achieved through interval sampling, pooling, or convolutional mapping with stride, reducing the number of spatial locations while preserving the channel features corresponding to each sampled region. The resulting downsampled feature map can express the regional distribution and local response of the feature map to be upsampled with a smaller spatial scale.

[0034] Conditional latent representations carry continuous semantic data encoded from conditional input information. To generate attention modulation information, spatial location features in the downsampled feature map can be converted into query features, and conditional latent representations can be converted into key and value features. The query features corresponding to each spatial location are matched with the key features to form matching data between conditional content and spatial regions. Then, the value features are weighted and aggregated using the matching data to form the modulation feature representation corresponding to each spatial location.

[0035] Attention modulation information is formed by organizing the modulation feature representations corresponding to each spatial location according to spatial arrangement. Attention modulation information preserves both the regional positional relationships in the downsampled feature map and the semantic responses in the conditional latent representation, enabling the conditional content to correspond to different spatial regions. Using downsampled feature maps in matching reduces the number of spatial features and decreases the amount of data processing required to form the modulation feature representation.

[0036] In one implementation, an interval sampling method is used to read channel features corresponding to multiple spatial locations from the feature map to be upsampled, and a downsampled feature map is formed according to the arrangement of the sampling positions. A query mapping layer processes the downsampled feature map, while a key mapping layer and a value mapping layer process the conditional latent representation respectively. The matching results between the query features and key features are normalized, and then the value features are aggregated based on the normalization results to form attention modulation information. This implementation is suitable for situations where the spatial scale of the feature map to be upsampled is large and the amount of data processing needs to be controlled.

[0037] In another implementation, a strided convolutional mapping is used to process the feature map to be upsampled, reducing the spatial scale while adjusting the feature channels to form a downsampled feature map. The conditional latent representation can be decomposed into multiple semantic unit features. Each spatial location in the downsampled feature map is matched with multiple semantic unit features, and a modulation feature representation for the corresponding spatial location is formed based on the matching results. Multiple modulation feature representations are arranged spatially to form attention modulation information. This implementation is suitable for situations where the conditional content contains multiple object attributes or multiple region descriptions.

[0038] This embodiment reduces the number of spatial features involved in condition matching by downsampling the feature map to be upsampled, while preserving the regional distribution and channel response. By matching the spatial features in the downsampled feature map with the semantic data in the conditional latent representation and forming attention modulation information, each spatial region can obtain a modulation expression corresponding to the condition content, thereby reducing the deviation between the local content of the image and the meaning of the input condition.

[0039] S50, modulate the dynamic filter bank based on the attention modulation information to generate a modulated dynamic filter bank, and upsample the feature map to be upsampled based on the modulated dynamic filter bank to generate an upsampled feature map. In this embodiment, the attention modulation information includes modulation feature data corresponding to spatial locations, and the dynamic filter bank includes filter parameter data for feature mapping. During modulation, the correspondence between the modulation feature data and the filter parameter data needs to be established first. The modulation feature data can be mapped to modulation parameter data with dimensions consistent with the dynamic filter bank, according to the arrangement of filter spatial locations, input channels, and output channels. The mapping process can employ linear transformation, channel expansion, or spatial rearrangement to convert the modulation feature data from a regional semantic representation to a filter parameter representation. Then, the modulation parameter data is combined item by item with the filter parameters in the dynamic filter bank to form the modulated dynamic filter bank. Item-by-item combination can be achieved through parameter superposition or a combination of parameter scaling and parameter offset, ensuring that the modulation information corresponding to different spatial regions is reflected in the changes in the filter parameters.

[0040] The feature map to be upsampled retains the spatial content distribution and channel response before upsampling. When upsampling using a modulated dynamic filter bank, local feature regions need to be extracted around each spatial location in the feature map to be upsampled. Then, the modulated dynamic filter bank is used to convolve and map these local feature regions to form the channel features to be rearranged. The channel features to be rearranged retain the mapping result corresponding to the current spatial location, and also include the output channel data used to expand the spatial resolution. To obtain the upsampled feature map, the channel features to be rearranged need to be spatially rearranged according to the correspondence between the output channel and the target spatial location, so that the subdivided spatial information originally located in the channel dimension is transferred to a higher resolution spatial location. After rearrangement, the high-resolution features corresponding to each local region are written into the new spatial location, forming the upsampled feature map. The upsampled feature map formed in this way is affected by both the content distribution of the feature map to be upsampled and the attention modulation information, and can retain conditionally related local response differences when expanding the resolution.

[0041] This embodiment maps attention modulation information to modulation parameter data corresponding to a dynamic filter bank and combines it with filtering parameters item by item. The filtering parameters can change with the spatial semantic response. By using the modulated dynamic filter bank to perform convolution mapping on local feature regions and rearranging the channel features to be rearranged into a higher resolution spatial distribution, the upsampled feature map can retain the modulation differences corresponding to different regions during the resolution improvement process, reducing the situation of monotonous content response when expanding local details.

[0042] S60, the upsampled feature map is converted into a generated image.

[0043] In this embodiment, the upsampled feature map already has a high spatial resolution, but the internal data still exists in the form of feature channels and has not yet been converted into image data that can be directly displayed or stored. During the conversion, the output mapping unit needs to perform channel compression and image channel reorganization on the upsampled feature map, so that the semantic response, texture response, and boundary response in the feature channels are converged into image channel data. The output mapping unit can use convolutional mapping, linear mapping, or position-by-position channel transformation to transform the multi-channel feature values ​​corresponding to each spatial position into image channel feature values. The image channel feature values ​​are used to represent the brightness information, color information, or grayscale information of the generated image at the corresponding spatial position.

[0044] The generation of the image also involves numerical constraints and spatial restoration. Numerical constraints are used to adjust the image channel feature values ​​to fit within the data range of the target image. This can be achieved through interval compression, amplitude cropping, or normalization mapping, ensuring that the image channel feature values ​​output at different spatial locations meet the image writing requirements. Spatial restoration organizes the image channel feature values ​​according to their spatial location and the arrangement of image channels, combining the channel data at each spatial location into a complete image. For multi-channel images, the image channels can be written to each spatial location in sequence; for single-channel images, the output values ​​corresponding to each spatial location can be directly organized into a grayscale image. After channel mapping, numerical constraints, and spatial restoration, the upsampled feature map is converted into the generated image.

[0045] In some implementations, a detail correction unit can be added. This unit balances the local high-frequency responses in the image channel eigenvalues, reducing edge overshoot, local color block breaks, or fine texture distortion, thus maintaining the visual continuity of the generated image. Detail correction can be performed after image channel mapping or in conjunction with numerical constraints.

[0046] This embodiment maps the upsampled feature map to image channels, converting the multi-channel response in the feature space into image channel data. By applying numerical constraints to the mapping results, the generated image meets the data range required for writing and display. By organizing the output data according to spatial location and image channel arrangement, the generated image can maintain spatial distribution and local detail expression, thereby reducing boundary distortion and channel mixing that occur during the conversion of feature data to image data.

[0047] In one embodiment, step S10 above includes: S101, Obtain the generated input information and conditional input information, input the generated input information into the input layer of the generation network, and use the input layer to map the generated input information into an initial feature representation; S102, Arrange the initial feature representations according to spatial location and feature channels to generate an initial feature map; S103, parse the conditional input information and generate a sequence of conditional data units arranged in the input order; S104, the conditional data unit sequence is input into a conditional coding network. The conditional coding network includes a set of conditional embedding parameters and multiple attention coding units connected sequentially along the feature propagation direction. The conditional embedding vectors corresponding to each conditional data unit in the conditional data unit sequence are extracted based on the set of conditional embedding parameters. The conditional embedding vectors are combined according to the order of the conditional data unit sequence to generate a conditional embedding sequence. S105, the conditional embedding sequence is input into the attention encoding unit located at the beginning of the feature transmission direction, and the conditional embedding sequence is feature-encoded using the attention encoding unit located at the beginning of the feature transmission direction to generate conditional encoded features; S106, along the feature transmission direction, input the conditional coding features generated by the previous attention coding unit into the next attention coding unit for feature coding, and determine the conditional coding features generated by the attention coding unit located at the end of the feature transmission direction as conditional sequence features. S107, aggregate the condition sequence features along the arrangement direction of the condition data unit sequence to generate a conditional latent representation.

[0048] In this embodiment, the generation input information carries the basic content data of the image to be generated, while the conditional input information carries the semantic data that constrains the generation result. When acquiring the generation input information and conditional input information, the corresponding data can be read from the request message, buffer, or storage area, and the data format can be standardized. The generation input information can be in vector form, multi-channel array form, or tensor form after pre-processing; the conditional input information can be text fragments, category tags, attribute fields, or a data set formed by combining various conditional contents. Setting these two types of data separately ensures that the source of image content and the source of conditional constraints remain independent, facilitating the subsequent formation of spatial content representation and conditional semantic representation respectively.

[0049] The input layer of the generative network transforms the generated input information into an initial feature representation suitable for internal processing within the image generation network. The input layer can contain linear mapping units, convolutional mapping units, or channel expansion units to adjust the dimension, number of channels, and data distribution of the generated input information. The mapped initial feature representation no longer maintains the single permutation of the original input but is transformed into an intermediate representation composed of multiple feature values, each set of feature values ​​corresponding to the local content information required for subsequent spatial expansion. If the generated input information itself is a one-dimensional vector, the input layer can first map the vector to an intermediate vector of a predetermined length, and then split the intermediate vector into multiple channel segments; if the generated input information itself already has local structure, the input layer can adjust the channel depth and preserve local structural relationships through convolutional mapping.

[0050] Spatial location and feature channels are used to organize the initial feature representation. Spatial location represents the arrangement relationship of different regions in the subsequent image, while feature channels represent the storage dimension of different content responses at the same spatial location. When arranging the initial feature representation according to spatial location and feature channels, the initial feature representation can be rearranged into a three-dimensional or four-dimensional tensor structure based on a predetermined width, height, and number of channels. This maintains a clear adjacency relationship between adjacent spatial locations, allowing multi-dimensional content responses at the same spatial location to be written into the corresponding channels. After forming the initial feature map, the image content data is transformed from a discrete input form into a spatially structured representation, facilitating subsequent unified processing of local regions and the overall structure.

[0051] Before forming a conditional latent representation, conditional input information needs to be parsed into a sequence of conditional data units. The parsing process identifies the smallest semantic units within the conditional input information and arranges them according to the input order. For text-based conditional input information, it can be segmented into multiple conditional data units at the word, sub-word, or character level. For conditional input information in the form of a combination of category tags and attribute fields, each field value can be considered a conditional data unit, and a sequence can be formed based on the order of field appearance. For mixed-form conditional input information, text units and field units can be mapped to the same sequence structure. The sequence of conditional data units preserves the permutation and combination relationships in the original conditional information, enabling the subsequent encoding process to perceive the positional relationships and semantic associations between different conditional data units.

[0052] Conditional coding networks handle conditional semantic extraction and context fusion. A set of conditional embedding parameters maps discrete conditional data units to continuous vectors. The conditional embedding vectors corresponding to each conditional data unit are read from or generated by the set of conditional embedding parameters, and then combined into a conditional embedding sequence according to the order of the conditional data unit sequence. Each vector in the conditional embedding sequence carries the initial semantic information of the corresponding conditional data unit, and multiple vectors together constitute a serialized representation of the entire conditional content. Multiple attention coding units are connected sequentially along the feature propagation direction. Each attention coding unit models the relationships between different positions within the sequence, enabling the semantic representation of a conditional data unit to be updated by incorporating information from preceding and following conditional data units. The attention coding unit at the beginning of the feature propagation direction receives the conditional embedding sequence and outputs conditional encoded features. Subsequent attention coding units continue to iteratively update the conditional encoded features, and the attention coding unit at the end of the feature propagation direction outputs conditional sequence features. The resulting conditional sequence features simultaneously preserve the local semantics of individual conditional data units and the combined semantics of the entire conditional content.

[0053] After aggregating the conditional sequence features along the arrangement direction of the conditional data unit sequence, a conditional latent representation is formed. Aggregation can be performed using methods such as average aggregation, weighted aggregation, or fusion after selecting features at specific locations. The aim is to compress the conditional expression in sequence form into a continuous vector of fixed dimension. The conditional latent representation no longer corresponds to a single conditional data unit, but rather to a comprehensive semantic expression of the entire conditional input information. The fixed-dimensional conditional latent representation facilitates interfaceing with subsequent image feature processing modules and maintains a consistent data interface across conditional input information of different lengths. An initial feature map is formed through the input layer of the generative network, and the conditional latent representation is formed through the conditional encoding network. Image content data and conditional semantic data undergo structured expression and semantic compression, respectively, enabling subsequent processing to read both spatial content and conditional constraint information.

[0054] This embodiment maps and rearranges the generated input information into an initial feature map through an input layer, resulting in generated content with clear spatial and feature channel relationships, facilitating the preservation of regional distribution information in subsequent processing. By parsing the conditional input information into a sequence of conditional data units and utilizing a set of conditional embedding parameters and multiple attention encoding units to form a conditional latent representation, the local and global semantics of the conditional content can be compressed into a unified, continuous expression. Thus, image content representation and conditional semantic representation each form stable data results, reducing the likelihood that conditional information will struggle to fully participate in content adjustment during subsequent generation.

[0055] In one embodiment, step S20 above includes: S201, the initial feature map is input into a feature transformation network, the feature transformation network comprising a plurality of residual feature transformation units connected sequentially along the feature transformation propagation direction; S202, the initial feature map is determined as the input feature map of the residual feature transformation unit located at the beginning of the feature transformation transmission direction. The input feature map is convolved and mapped using the residual feature transformation unit located at the beginning of the feature transformation transmission direction to generate a mapped feature map. The input feature map and the mapped feature map are superimposed according to their corresponding spatial positions and corresponding feature channels to generate an output feature map. S203, along the feature transformation propagation direction, the output feature map generated by the previous residual feature transformation unit is determined as the input feature map of the next residual feature transformation unit. The input feature map is convolved and mapped using the next residual feature transformation unit to generate a mapped feature map. The input feature map and the mapped feature map are superimposed according to their corresponding spatial positions and corresponding feature channels to generate an output feature map. S204, the output feature map generated by the residual feature transformation unit located at the end of the feature transformation transmission direction is determined as the feature map to be upsampled; S205, average and aggregate the feature values ​​of each spatial location corresponding to each feature channel in the feature map to be upsampled to generate channel statistical feature values ​​corresponding to each feature channel, arrange the channel statistical feature values ​​according to the feature channel arrangement order, and generate global feature statistics. S206, the conditional latent representation and the global feature statistics are concatenated along the feature dimension to generate a feature representation using a filter.

[0056] In this embodiment, the initial feature map carries spatial content information after pre-mapping. The feature transformation network is used to reshape the feature responses within and between channels while maintaining the spatial layout relationship. The feature transformation network consists of multiple residual feature transformation units connected sequentially along the feature transformation propagation direction. The output of the previous unit serves as the input of the next unit, enabling the features to be refined step by step during continuous propagation. Using multiple residual feature transformation units in series helps to preserve the original spatial distribution information while enhancing semantic expression layer by layer, avoiding excessive weakening of local responses after multiple convolutional mappings.

[0057] The residual feature transformation unit, located at the beginning of the feature transformation propagation direction, receives the initial feature map as the input feature map. This residual feature transformation unit extracts local texture information, boundary responses, and region combination relationships from the input feature map through convolutional feature mapping, forming a mapped feature map. The result of the convolutional feature mapping is then superimposed on the input feature map item by item according to spatial location and feature channels. The superimposed result is used as the output feature map. This item-by-item superposition ensures that the new features formed by the convolutional mapping and the original content in the input feature map are preserved simultaneously. The convolutional mapping focuses on supplementing and strengthening the expression of local content, while the input feature map focuses on preserving existing spatial information. The combined output feature map possesses both the original structural information and the transformed semantic information.

[0058] Subsequent residual feature transformation units continue processing along the same propagation direction. The output feature map generated by the previous residual feature transformation unit is determined as the input feature map of the next residual feature transformation unit, which then performs convolutional feature mapping to form a new mapped feature map. The input feature map and the new mapped feature map are then superimposed item by item according to spatial location and feature channel to form a new output feature map. Multiple residual feature transformation units repeat this operation process, allowing the feature map to absorb the combined information of local regions layer by layer during propagation, while continuously retaining the existing content expression. The output feature map generated by the residual feature transformation unit located at the end of the feature transformation propagation direction is determined as the feature map to be upsampled. The feature map to be upsampled thus forms a relatively sufficient feature expression, containing both the response of the local spatial region and the semantic information accumulated after multiple transformations.

[0059] Global feature statistics are extracted from the feature map to be upsampled and are used to describe the overall distribution of the feature map across all feature channels. During extraction, the feature values ​​of each feature channel are averaged and aggregated across all spatial locations, using the feature channel as the statistical unit, to obtain the channel statistical feature value for that channel. Average aggregation compresses the response of the same channel at different spatial locations into a single statistical result, thus providing a centralized expression of the overall activity of that channel across the entire feature map. All channel statistical feature values ​​are then organized sequentially according to the feature channel arrangement to form the global feature statistics. The global feature statistics no longer preserve spatial location differences but instead summarize spatial distribution information into channel-level statistical results, reflecting the overall content state of the feature map to be upsampled.

[0060] The conditional latent representation carries the semantic expression of conditional information, while the global feature statistics carry the overall content expression of the feature map to be upsampled. Concatenating these two along the feature dimension forms a filter-generated feature representation. The concatenated data simultaneously contains both conditional semantic information and image content statistics, ensuring that the feature representation used subsequently is not limited to a single source. The conditional latent representation provides the constraints of the generation target, while the global feature statistics provide the overall state of the current feature map. The combined data result can simultaneously reflect the generation requirements and content distribution, facilitating the subsequent construction of parameter expressions associated with the conditional content and the current feature state.

[0061] This embodiment continuously transforms the initial feature map through multiple residual feature transformation units, retaining the original content of the input feature map after each level of convolutional feature mapping. The upsampled feature map can enhance local semantic expression while maintaining spatial layout. By averaging and aggregating the upsampled feature map by feature channel, a global feature statistic is formed, providing a centralized representation of the overall content state of the entire feature map. The conditional latent representation and the global feature statistic are then concatenated along the feature dimension. The filter-generated feature representation carries both conditional semantic information and content statistics, which helps the subsequent parameter formation process to take into account both generation constraints and the current content distribution, reducing information bias caused by relying solely on local features or solely on conditional semantics.

[0062] In one embodiment, step S30 above includes: S301, Obtain filter configuration data, normalized scaling parameters, and normalized offset parameters. The filter configuration data includes filter spatial arrangement information, input channel arrangement information, and output channel arrangement information. S302, the filter generation feature representation is input into the filter generation network, the filter generation network including a plurality of parameter mapping units connected sequentially along the parameter mapping transfer direction; S303, the parameter mapping unit located at the beginning of the parameter mapping transmission direction is used to perform parameter mapping and nonlinear activation on the filter generation feature representation to generate parameter mapping features; S304, along the parameter mapping transmission direction, input the parameter mapping features generated by the previous parameter mapping unit into the next parameter mapping unit for parameter mapping and nonlinear activation, and determine the parameter mapping features generated by the parameter mapping unit located at the end of the parameter mapping transmission direction as the filter parameter representation; S305, based on the filter space arrangement information, the input channel arrangement information and the output channel arrangement information, the filter parameter representation is rearranged and reorganized to generate a candidate dynamic filter bank; S306, extract the parameters of each candidate filter in the candidate dynamic filter bank, square each candidate filter parameter, accumulate and take the square root of each squared candidate filter parameter to generate the filter normalization quantity. S307, using the normalized filter value as the divisor, perform division on each candidate filter parameter to generate the normalized filter parameters corresponding to each candidate filter parameter. S308, multiply each normalized filter parameter with the normalized scaling parameter, and add the multiplication result with the normalized offset parameter to generate dynamic filter parameters. Organize the dynamic filter parameters according to the filter space arrangement information, the input channel arrangement information and the output channel arrangement information to generate a dynamic filter bank.

[0063] In this embodiment, the filter generation feature representation simultaneously includes conditional semantic information and content statistics. When generating a dynamic filter bank, the organization rules of the filter parameters need to be determined first. Filter configuration data is used to specify the arrangement of filter parameters in the spatial dimension, input channel dimension, and output channel dimension. Specifically, the filter spatial arrangement information is used to define the parameter layout at each spatial location within a single filter, the input channel arrangement information is used to define the parameter distribution on different input channels, and the output channel arrangement information is used to define the parameter grouping method used for different output results. Normalization scaling parameters and normalization offset parameters are used to adjust the normalized parameter amplitudes and parameter references, ensuring that the generated filter parameters maintain a controllable numerical distribution.

[0064] After receiving the filter-generated feature representation, the filter generation network performs parameter transformation step by step along the parameter mapping propagation direction. The parameter mapping unit at the beginning of the parameter mapping propagation direction performs parameter mapping on the filter-generated feature representation, transforming the original feature data from a semantic expression form into intermediate feature data that more closely resembles the distribution of filter parameters. Nonlinear activation is used to change the response distribution after linear mapping, allowing feature differences caused by different conditions and statistical states of different content to be further amplified. After passing through the first parameter mapping unit, parameter-mapped features are formed, which already possess preliminary parameter representation capabilities.

[0065] Subsequent parameter mapping units continue processing the parameter mapping features generated by the previous unit along the parameter mapping propagation direction. Each subsequent parameter mapping unit performs parameter mapping and nonlinear activation on the input parameter mapping features, gradually converging the parameter representation from a coarse intermediate state to a parameter state suitable for filter construction. As multi-level mapping continues, semantic and content differences in the filter-generated feature representation are gradually transferred into the parameter space. The parameter mapping features output by the last parameter mapping unit in the parameter mapping propagation direction are determined as the filter parameter representation. The filter parameter representation itself still exists in a one-dimensional or low-structured form and has not yet formed a directly usable filter arrangement structure.

[0066] Based on the filter spatial arrangement, input channel arrangement, and output channel arrangement, the filter parameter representations are rearranged and reorganized to form a candidate dynamic filter bank. The purpose of this rearrangement is to write consecutively arranged parameter values ​​into predetermined spatial, input, and output channel positions, transforming the parameter values ​​from a linear distribution state to a filter structure state. After forming the candidate dynamic filter bank, each candidate filter parameter is assigned a clear spatial and channel meaning, allowing subsequent processing to directly read these parameters according to the filter structure.

[0067] After the candidate dynamic filter bank is formed, all candidate filter parameters need to be normalized. The normalized quantity is obtained by squaring each candidate filter parameter, summing the squared results, and then taking the square root of the sum. The squaring process converts parameter values ​​with different signs into positive amplitude contributions, the summation process forms the total amplitude of all parameters, and the square root process restores the total to a numerical scale consistent with the original parameter dimensions. The resulting normalized filter quantity reflects the overall parameter amplitude of the candidate dynamic filter bank.

[0068] Normalized filter parameters are obtained by dividing each candidate filter parameter by the filter normalization factor. This division unifies the parameter sets generated under different input conditions into a comparable numerical range, preserving relative differences between parameters and compressing overall amplitude fluctuations. The normalized filter parameters are then multiplied by the normalization scaling parameter to adjust the parameter response intensity; the result is then added to the normalization offset parameter to adjust the parameter distribution center. These multiplications and additions form the dynamic filter parameters. The dynamic filter parameters are then reorganized according to the filter space arrangement, input channel arrangement, and output channel arrangement to obtain the dynamic filter bank. This dynamic filter bank now possesses both parameter structure and numerical stability, reflecting both conditional and content differences in the filter generation feature representation.

[0069] For example, the formula for generating candidate dynamic filter banks based on filter generation feature representation is:

[0070] in, This represents the feature map to be upsampled. This represents the global feature statistics extracted from the feature map to be upsampled. It is implemented by averaging and aggregating the feature values ​​of the same feature channel along each spatial location. ⊕ represents the conditional latent representation, used to carry continuous semantic data formed by encoding conditional input information. ⊕ indicates concatenation along the feature dimension; the concatenation result corresponds to the feature representation generated by the filter. Let θ represent the filter generation network, and let θ represent the set of parameters in the filter generation network. This represents a candidate dynamic filter bank.

[0071] The formula for organizing the spatial channels of a candidate dynamic filter bank is:

[0072] in, This represents the candidate dynamic filter bank. k represents the filter space size corresponding to the filter space arrangement information, which is used to determine the spatial range of the convolutional feature mapping in the local feature region. This indicates the number of input feature channels corresponding to the input channel arrangement information. This indicates the number of output feature channels corresponding to each spatial location after upsampling. r represents the upsampling factor. This represents the number of output channels formed by the convolutional feature mapping before spatial rearrangement, where different output channel features correspond to different spatial positions after upsampling.

[0073] The parameters in the candidate dynamic filter bank can be normalized to ensure that the filter parameters generated by different filter feature representations have a comparable data range in terms of overall amplitude. Filter normalization is expressed by the following formula:

[0074] in, This represents a candidate dynamic filter bank. This represents the filter normalization quantity, and its formation process corresponds to squaring, accumulating, and taking the square root of the candidate dynamic filter parameters. β represents the normalization scaling parameter, used to adjust the response amplitude of the normalized filter parameters. β represents the normalization offset parameter, used to adjust the position of the scaled parameter distribution. This represents a dynamic filter bank, corresponding to the result after normalization, scaling, and offset processing.

[0075] When training the filter generation network, the training loss forms gradient data for the parameter set in the filter generation network. To limit the magnitude of the gradient data during a single parameter update, a gradient clipping constraint can be applied to the gradient data. Gradient clipping is expressed by the following formula:

[0076] in, This represents the loss value generated during training. θ represents the set of updatable parameters in the filter generation network, including the mapping parameters corresponding to multiple parameter mapping units. This indicates that the loss value is applied to the gradient data formed by the set of updatable parameters. This represents the L2 norm of the gradient data, reflecting the overall gradient magnitude corresponding to this parameter update. The value 0.1 indicates the gradient magnitude threshold shown in the figure. When the L2 norm of the gradient data exceeds this threshold, the gradient data can be reduced proportionally, and then the reduced gradient data can be used to update the filter and generate the parameter set in the network.

[0077] This embodiment employs a filter generation network to perform multi-level parameter mapping and nonlinear activation on the filter-generated feature representation. Conditional semantic information and content statistics are progressively transformed into filter parameter expressions. By rearranging and recombining the filter parameter representation based on the filter spatial arrangement, input channel arrangement, and output channel arrangement, a candidate dynamic filter bank with a clear spatial and channel structure is generated. Through squaring, accumulating, square rooting, dividing, multiplying, and adding the candidate filter parameters, the overall amplitude and distribution center of the dynamic filter parameters are adjusted. The resulting dynamic filter bank can change with the input content while maintaining a relatively stable parameter distribution, which is beneficial for reflecting both conditional and content differences during subsequent feature mapping.

[0078] In one embodiment, step S40 above includes: S401, obtain query projection parameters, key projection parameters and value projection parameters, determine the interval sampling position from the feature map to be upsampled, extract channel features from each interval sampling position, organize the channel features according to the spatial arrangement of each interval sampling position, and generate a downsampled feature map; S402, based on the query projection parameters, channel projection is performed on the channel features corresponding to each spatial position in the downsampled feature map to generate query features corresponding to each spatial position, and the query features are arranged according to their spatial positions in the downsampled feature map to generate a query feature representation; S403, Based on the key projection parameters, feature projection is performed on the conditional latent representation to generate multiple key features arranged along the conditional feature dimension, and the key features are combined to generate a key feature representation; based on the value projection parameters, feature projection is performed on the conditional latent representation to generate multiple value features arranged along the conditional feature dimension, and the value features are combined to generate a value feature representation. S404, for the query features corresponding to each spatial location in the query feature representation, perform dot product matching between the query features and each key feature respectively to generate the attention matching representation corresponding to each spatial location on the condition feature dimension; S405, normalize the attention matching representation corresponding to each spatial location along the conditional feature dimension to generate the spatial attention weight corresponding to each spatial location on the conditional feature dimension. S406, according to the spatial attention weights corresponding to each spatial location on the condition feature dimension, weighted aggregation of each value feature is performed to generate a modulation feature representation corresponding to each spatial location. The modulation feature representations are arranged according to their spatial positions in the downsampled feature map to generate attention modulation information.

[0079] In this embodiment, the feature map to be upsampled contains both spatial location dimension and feature channel dimension. The spatial location dimension reflects the arrangement relationship of different regions, while the feature channel dimension reflects the differences in texture, boundary, structure, and semantic response of each region. To form a compact representation suitable for conditional matching, the interval sampling positions need to be determined first. The interval sampling positions can be selected horizontally and vertically according to a predetermined step size, or they can be selected based on a predetermined sampling grid. After reading the feature values ​​of all channels for each interval sampling position, the channel features corresponding to that position are obtained. These channel features are then organized according to the spatial arrangement of each sampling position to form a downsampled feature map. The resulting data reduces the spatial scale while preserving the regional distribution relationship and the main channel responses of each region.

[0080] The query projection parameters are used to transform the channel features corresponding to each spatial location in the downsampled feature map into query features suitable for matching. During channel projection, the channel features at each spatial location are interacted with the query projection parameters to obtain the query features corresponding to that spatial location. All query features are arranged in order of their spatial locations in the downsampled feature map to form a query feature representation. The query feature representation preserves the independence between spatial locations, allowing for subsequent position-by-position matching with conditional semantics.

[0081] The conditional latent representation carries comprehensive semantic information about the conditional content. Key projection parameters and query projection parameters serve different purposes. Key projection parameters are used to split or expand the conditional latent representation into multiple key features arranged along the conditional feature dimension, allowing the conditional content to participate in matching as multiple semantic components. Value projection parameters are used to convert the conditional latent representation into multiple value features corresponding to these semantic components, enabling the conditional content to participate in modulation output as aggregable data after matching. All key features are combined to form the key feature representation, and all value features are combined to form the value feature representation. The key feature representation reflects the matching reference for different conditional semantic components, while the value feature representation reflects the output content of different conditional semantic components after matching.

[0082] For any spatial location, dot product matching is performed between the query feature and each key feature to obtain the attention-matched representation of that spatial location along the conditional feature dimension. The result of the dot product matching reflects the strength of the correspondence between the content response of the current spatial location and each conditional semantic component. Since a spatial location may be associated with multiple conditional semantic components to varying degrees, the attention-matched representation retains a set of matching results expanded along the conditional feature dimension, rather than a single matching value. This allows the relationship between spatial location and conditional semantics to remain granular.

[0083] The normalization process is performed along the conditional feature dimension, aiming to transform the attention matching representations at each spatial location into comparable weight distributions. The resulting spatial attention weights after normalization correspond to the degree of response of the same spatial location to different conditional semantic components. Because normalization is performed along the conditional feature dimension, multiple sets of weights corresponding to the same spatial location form a relative proportional relationship, thus distinguishing which type of conditional semantic component the current region is closer to, and also reflecting the degree of joint influence of multiple conditional semantic components in that region.

[0084] After obtaining the spatial attention weights, the value feature representations are weighted and aggregated. For each spatial location, the spatial attention weights of that location on the conditional feature dimension are used to sum the value features in a weighted manner, forming the modulation feature representation corresponding to that spatial location. The modulation feature representation combines the regional location response and the conditional semantic response. After rearranging the modulation feature representations corresponding to all spatial locations in the spatial location order of the downsampled feature map, attention modulation information is obtained. The attention modulation information retains both the spatial arrangement relationship in the downsampled feature map and the semantic differences decomposed from the conditional latent representation, thus reflecting the response results of different spatial regions to different conditional semantic components.

[0085] For example, the formula for the size of the downsampled feature map formed by interval sampling is:

[0086] in, This represents the downsampled feature map. h and w represent the dimensions of the feature map to be upsampled in two spatial directions, respectively. s represents the interval sampling rate, which can be twice the sampling rate. This indicates the number of feature channels retained before and after downsampling. This formula applies to cases where h and w can be completely divided according to the interval sampling ratio.

[0087] The formulas for generating query features, key features, and value features using attention modulation are as follows:

[0088]

[0089]

[0090] in, The query feature representation is formed by projecting the downsampled feature map onto the query. This indicates a query projection process implemented using single-position convolution mapping, which is a type of channel projection implementation. represents the downsampled feature map. K represents the key feature representation, formed by key projection of the conditional latent representation. V represents the value feature representation, formed by value projection of the conditional latent representation. Indicates the key projection parameters. Indicates the projection parameter. This represents a conditional latent representation.

[0091] The formula for forming spatial attention weights along the conditional feature dimension is:

[0092] in, This represents the spatial attention weight corresponding to the j-th conditional feature at the i-th spatial location. This represents the query feature corresponding to the i-th spatial location in the query feature representation. This represents the j-th key feature arranged along the conditional feature dimension in the key feature representation. This represents the dot product matching result between the query feature and the key feature. d represents the feature dimension used when matching the query feature and the key feature. Used to adjust the numerical scale of the dot product matching results. This represents the index of the conditional features participating in the normalization process for the same spatial location. The denominator represents the sum of the exponents of all matching results formed along the conditional feature dimension for the i-th spatial location.

[0093] This embodiment compresses the number of spatial units involved in matching by intermittently sampling the upsampled feature map to form a downsampled feature map, while preserving the regional distribution relationship and the response of the main channels. By using query projection parameters, key projection parameters, and value projection parameters to form query feature representation, key feature representation, and value feature representation respectively, the spatial region content and conditional semantic content can be mapped under a unified matching framework. Through dot product matching, normalization, and weighted aggregation, the resulting attention modulation information can be refined to each spatial location and reflect the differences in the response of each spatial location to different conditional semantic components. The resulting data results have both spatial location discrimination and conditional semantic discrimination capabilities, reducing the situation where insufficient semantic response is caused by using a single modulation content in different regions.

[0094] In one embodiment, step S50 above includes: S501, Obtain filter spatial arrangement information, input channel arrangement information and output channel arrangement information, and extract dynamic filter parameter set from the dynamic filter bank; S502, extract the modulation feature representations arranged in spatial position from the attention modulation information, aggregate the modulation feature representations according to the spatial position arrangement order of each modulation feature representation, and generate the filter modulation feature representation; S503, based on the filter spatial arrangement information, the input channel arrangement information and the output channel arrangement information, the filter modulation feature representation is converted into a filter modulation parameter set, the filter modulation parameter set is superimposed with the dynamic filter parameter set to generate a modulated dynamic filter parameter set, and the modulated dynamic filter parameter set is organized according to the filter spatial arrangement information, the input channel arrangement information and the output channel arrangement information to generate a modulated dynamic filter group; S504, determine the spatial location to be processed in the feature map to be upsampled, extract local feature regions from each spatial location to be processed, use the modulated dynamic filter bank to perform convolutional feature mapping on each local feature region, generate channel features to be rearranged corresponding to each spatial location to be processed, arrange each channel features to be rearranged according to each spatial location to be processed, and generate a feature map to be rearranged. S505, Based on the output channel arrangement information, generate the upsampling position correspondence between each output channel feature in the channel features to be rearranged and the upsampling spatial position corresponding to each spatial position to be processed in the feature map to be rearranged; S506, based on the upsampling position correspondence, rearrange the output channel features in the channel features to be rearranged corresponding to each spatial position in the feature map to be rearranged to the corresponding upsampling spatial position to generate an upsampling feature map.

[0095] In this embodiment, the filter spatial arrangement information is used to record the arrangement position of the filter parameters within the local sampling region, the input channel arrangement information is used to record the correspondence between the channel data and the filter parameters in the local feature region, and the output channel arrangement information is used to record the output channel structure formed after convolution mapping. When reading the dynamic filter bank, the dynamic filter parameter set is extracted according to the above arrangement information, so that each dynamic filter parameter has a definite spatial position, input channel position, and output channel position, which facilitates the subsequent writing of modulation data to the corresponding parameter positions.

[0096] Attention modulation information comprises multiple modulation feature representations organized spatially, each carrying the matching result between the corresponding region and the conditional semantics. These modulation feature representations are aggregated according to their spatial order, either through sequential concatenation or by preserving the position index, to form filter modulation feature representations. These filter modulation feature representations retain the modulation data and their arrangement relationships corresponding to different regions, enabling subsequent parameter transformations to identify the spatial region from which the modulation data originates.

[0097] The filter modulation feature representation needs to be converted into a filter modulation parameter set with a corresponding dimension to the dynamic filter parameter set. During the conversion, the regional modulation data can be allocated to filter space positions based on the filter space arrangement information, expanded to the parameter positions corresponding to each input channel based on the input channel arrangement information, and written to the parameter positions corresponding to each output channel based on the output channel arrangement information. The filter modulation parameter set and the dynamic filter parameter set are superimposed according to the corresponding parameter positions to generate the modulated dynamic filter parameter set. The superimposed parameters simultaneously contain the original content response of the dynamic filter and the semantic response brought by the attention modulation information. The modulated dynamic filter parameter set is then organized according to the three types of arrangement information to form a modulated dynamic filter bank.

[0098] The spatial locations to be processed in the feature map to be upsampled are used to determine the local feature reading region. For each spatial location to be processed, the feature values ​​corresponding to adjacent locations can be read according to the filtering space range of the modulated dynamic filter bank to form a local feature region. When performing convolutional feature mapping on the local feature region using the modulated dynamic filter bank, the input channel arrangement information controls the channel correspondence between the local features and the filtering parameters, and the output channel arrangement information controls the channel organization of the mapping result. After convolutional feature mapping, each spatial location to be processed forms a channel feature to be rearranged. All channel features to be rearranged are arranged according to the spatial locations to be processed to form a feature map to be rearranged.

[0099] The output channel features in the feature map to be rearranged are used to write the spatial locations after resolution enhancement. Based on the output channel arrangement information, different upsampling spatial locations can be assigned to the output channel features corresponding to the same spatial location to be processed, forming an upsampling position correspondence. During rearrangement, the output channel features corresponding to each spatial location to be processed are read from the feature map to be rearranged, and written to the corresponding upsampling spatial locations according to the upsampling position correspondence. This transforms the subdivided region data originally stored in the channel dimension into high-resolution feature data in the spatial dimension, forming an upsampling feature map.

[0100] For example, the modulated filter index corresponding to the upsampling spatial location is expressed as:

[0101] in, This represents the modulated dynamic filter bank. x and y represent the spatial location indices in the upsampled feature map. r represents the upsampling factor. and This indicates the position index of the current upsampling spatial location within the local upsampling region, used to determine the modulated filter parameters and output channel characteristics that should be read.

[0102] The formula for forming the modulated dynamic filter bank from the attention-modulated information is:

[0103] in, This represents the modulated dynamic filter bank. This represents a dynamic filter bank. This represents the spatial attention weight corresponding to the j-th conditional feature at the i-th spatial location. This represents the j-th value feature arranged along the conditional feature dimension in the value feature representation. After aggregation, a filter modulation feature representation is formed. This represents the parameter transformation process, which converts the filter modulation feature representation into a set of filter modulation parameters based on the filter spatial arrangement information, input channel arrangement information, and output channel arrangement information. The addition operation represents the corresponding superposition of the filter modulation parameter set and the dynamic filter parameter set.

[0104] This embodiment converts attention modulation information into a set of filter modulation parameters corresponding to the dynamic filter parameter set, and then superimposes these parameters onto the dynamic filter parameter set. The modulated dynamic filter bank can combine the region content response and the conditional semantic response. By performing convolutional feature mapping on local feature regions in the upsampled feature map using the modulated dynamic filter bank, and rearranging the output channel features to the upsampled spatial position according to the output channel arrangement information, the spatial resolution enhancement process can preserve the modulation differences corresponding to different regions, thereby reducing the inconsistency between high-resolution local details and conditional semantics.

[0105] In one embodiment, step S60 above includes: S601, the upsampled feature map is used as the final upsampled feature map to be received in the final upsampled stage, the final global feature statistics are extracted from the final upsampled feature map, and the conditional latent representation and the final global feature statistics are combined into the final filter to generate feature representation. S602, in the final stage upsampling stage, a final stage dynamic filter bank is generated based on the feature representation generated by the final stage filter, the final stage feature map to be upsampled is downsampled to generate a final stage downsampled feature map, and final stage attention modulation information is generated based on the final stage downsampled feature map and the conditional latent representation; S603, in the final-stage upsampling stage, the final-stage dynamic filter bank is modulated based on the final-stage attention modulation information to generate a final-stage modulated dynamic filter bank, and the final-stage upsampled feature map is upsampled based on the final-stage modulated dynamic filter bank to generate a final-stage upsampled feature map. S604, the final-level upsampled feature map is passed to the output layer, the output layer includes image channel mapping parameters, and the final-level upsampled feature map is mapped to the image channel based on the image channel mapping parameters to generate an image channel feature map; S605, arrange the feature values ​​of each channel according to the spatial position and image channel in the image channel feature map to obtain the generated image.

[0106] In this embodiment, the upsampled feature map already has a high spatial resolution at the current stage, but the data in each spatial location still exists in the form of multi-channel features and has not yet been converted into image channel data. Using the upsampled feature map as the final upsampled feature map received in the final upsampled stage allows the final image formation process to retain the spatial distribution relationship established in the previous stage. The final-level global feature statistics are extracted from the final upsampled feature map and are used to aggregate the overall response of the entire feature map in each feature channel. During extraction, for each feature channel, the feature values ​​at all spatial locations can be averaged and aggregated to form the statistical value of the corresponding channel, and then organized into the final-level global feature statistics according to the feature channel order. The data formed in this way can reflect the overall content state of the final upsampled feature map. The conditional latent representation carries the comprehensive semantic information of the conditional content. The conditional latent representation is combined with the final-level global feature statistics to form the final-level filter generating feature representation, so that the parameter expression used in the final stage simultaneously contains conditional constraint information and the overall distribution information of the current image content.

[0107] The final upsampling stage continues to utilize the dynamic filter parameter formation mechanism. The final-stage filter generates a feature representation used to generate the final-stage dynamic filter bank, ensuring that the filter parameters correspond to the current conditional content and the overall image state. The final-stage upsampled feature map is downsampled to form a final-stage downsampled feature map, compressing its spatial scale while preserving representative content from local regions. The final-stage downsampled feature map and the conditional latent representation participate in matching to form final-stage attention modulation information. This final-stage attention modulation information records the response results of different spatial regions to each semantic component in the conditional content, allowing for refined parameter adjustment in the final stage down to different regions.

[0108] The final-stage attention modulation information is combined with the final-stage dynamic filter bank to form the final-stage modulated dynamic filter bank. The combination process can employ parameter-aligned superposition, scaling adjustment, or offset adjustment to incorporate modulation information corresponding to different spatial regions into the filter parameters. The final-stage modulated dynamic filter bank is then used for upsampling the final-stage feature map to be upsampled, ensuring that the final resolution improvement no longer depends solely on fixed parameters but simultaneously reflects the differences between conditional content and the current region content. After upsampling, the final-stage upsampled feature map is formed, where each spatial location already possesses a higher-resolution local feature representation.

[0109] After the final-stage upsampled feature map is passed to the output layer, it needs to be converted from multi-channel feature data to image channel data. The image channel mapping parameters in the output layer are used to establish the mapping relationship between feature channels and image channels. During mapping, the multi-channel feature values ​​at each spatial location can be compressed and linearly combined to form the image channel feature value corresponding to that spatial location. After all spatial locations are mapped, the image channel feature map is obtained. The spatial positions in the image channel feature map continue to follow the arrangement relationship in the final-stage upsampled feature map, and the image channel dimension is used to carry brightness information, grayscale information, or color information. After arranging the channel feature values ​​according to spatial position and image channel, the image channel feature map is reconstructed into the generated image. Thus, the high-resolution feature representation formed in the previous stage is converted into image data that can be directly displayed, stored, or transmitted.

[0110] This embodiment extracts final-level global feature statistics from the final-level feature map to be upsampled during the final-level upsampling stage, and combines them with the conditional latent representation to form a final-level filter-generated feature representation. The filtering parameters used in the final stage can simultaneously reflect the conditional content and the overall state of the current image. By forming final-level attention modulation information and modulating the final-level dynamic filter bank during the final-level upsampling stage, different spatial regions can obtain parameter responses corresponding to the conditional content during the final resolution improvement process. The final-level upsampled feature map is converted into an image channel feature map through the image channel mapping parameters in the output layer, and then recombined according to spatial location and image channels. The generated image can complete the image output while maintaining high-resolution local features, thereby reducing the deviation between the local details of the final image and the conditional content.

[0111] In one embodiment, an image generation apparatus based on a dynamic filter bank is provided, which corresponds one-to-one with the image generation method based on a dynamic filter bank described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the image generation device based on a dynamic filter bank of the present invention. The modules include an input characterization module 10, a filtering feature module 20, a filtering generation module 30, an attention modulation module 40, a modulation upsampling module 50, and an image output module 60. Detailed descriptions of each functional module are as follows: The input representation module 10 is used to acquire generated input information and conditional input information, convert the generated input information into an initial feature map, and encode the conditional input information into a conditional latent representation; The filtering feature module 20 is used to perform feature transformation on the initial feature map to generate a feature map to be upsampled, extract global feature statistics from the feature map to be upsampled, and combine the conditional latent representation and the global feature statistics into a filter to generate a feature representation. Filter generation module 30 is used to generate a dynamic filter bank based on the filter generation feature representation; Attention modulation module 40 is used to downsample the feature map to be upsampled to generate a downsampled feature map, and generate attention modulation information based on the downsampled feature map and the conditional latent representation. The modulation upsampling module 50 is used to modulate the dynamic filter bank based on the attention modulation information to generate a modulated dynamic filter bank, and to upsample the feature map to be upsampled based on the modulated dynamic filter bank to generate an upsampled feature map. Image output module 60 is used to convert the upsampled feature map into a generated image.

[0112] In one embodiment, the input characterization module 10 is specifically used for: Obtain the generated input information and conditional input information, input the generated input information into the input layer of the generation network, and use the input layer to map the generated input information into an initial feature representation; Arrange the initial feature representations according to their spatial location and feature channels to generate an initial feature map; The conditional input information is parsed to generate a sequence of conditional data units arranged in the input order; The conditional data unit sequence is input into a conditional coding network, which includes a set of conditional embedding parameters and multiple attention coding units connected sequentially along the feature propagation direction. The conditional embedding vectors corresponding to each conditional data unit in the conditional data unit sequence are extracted based on the set of conditional embedding parameters. The conditional embedding vectors are combined according to the order of the conditional data unit sequence to generate a conditional embedding sequence. The conditional embedding sequence is input into the attention encoding unit located at the beginning of the feature propagation direction, and the conditional embedding sequence is feature-encoded using the attention encoding unit located at the beginning of the feature propagation direction to generate conditional encoded features. Along the feature transmission direction, the conditional coding features generated by the previous attention coding unit are input into the next attention coding unit for feature encoding, and the conditional coding features generated by the attention coding unit located at the end of the feature transmission direction are determined as conditional sequence features. The conditional sequence features are aggregated along the arrangement direction of the conditional data unit sequence to generate a conditional latent representation.

[0113] In one embodiment, the filtering feature module 20 is specifically used for: The initial feature map is input into a feature transformation network, which includes multiple residual feature transformation units connected sequentially along the feature transformation propagation direction. The initial feature map is determined as the input feature map of the residual feature transformation unit located at the beginning of the feature transformation propagation direction. The input feature map is convolved and mapped using the residual feature transformation unit located at the beginning of the feature transformation propagation direction to generate a mapped feature map. The input feature map and the mapped feature map are superimposed according to their corresponding spatial positions and corresponding feature channels to generate an output feature map. Along the direction of feature transformation propagation, the output feature map generated by the previous residual feature transformation unit is determined as the input feature map of the next residual feature transformation unit. The input feature map is then convolved with the next residual feature transformation unit to generate a mapped feature map. The input feature map and the mapped feature map are then superimposed according to their corresponding spatial positions and corresponding feature channels to generate the output feature map. The output feature map generated by the residual feature transformation unit located at the end of the feature transformation transmission direction is determined as the feature map to be upsampled; The feature values ​​of each spatial location corresponding to each feature channel in the feature map to be upsampled are averaged and aggregated to generate channel statistical feature values ​​corresponding to each feature channel. The channel statistical feature values ​​are arranged according to the order of feature channel arrangement to generate global feature statistics. The conditional latent representation and the global feature statistics are concatenated along the feature dimension to generate a feature representation using a filter.

[0114] In one embodiment, the filter generation module 30 is specifically used for: Obtain filter configuration data, normalized scaling parameters, and normalized offset parameters. The filter configuration data includes filter spatial arrangement information, input channel arrangement information, and output channel arrangement information. The filter generation feature representation is input to the filter generation network, which includes a plurality of parameter mapping units connected sequentially along the parameter mapping transfer direction. The parameter mapping unit located at the beginning of the parameter mapping transmission direction is used to perform parameter mapping and nonlinear activation on the filter generation feature representation to generate parameter mapping features; Along the parameter mapping transmission direction, the parameter mapping features generated by the previous parameter mapping unit are input into the next parameter mapping unit for parameter mapping and nonlinear activation. The parameter mapping features generated by the parameter mapping unit located at the end of the parameter mapping transmission direction are determined as the filter parameter representation. Based on the filter space arrangement information, the input channel arrangement information, and the output channel arrangement information, the filter parameter representation is rearranged and reorganized to generate a candidate dynamic filter bank; Extract the parameters of each candidate filter in the candidate dynamic filter bank, square each candidate filter parameter, accumulate and take the square root of each squared candidate filter parameter to generate the filter normalization quantity. Using the filter normalization factor as the divisor, the parameters of each candidate filter are divided to generate the normalized filter parameters corresponding to each candidate filter parameter. Each normalized filter parameter is multiplied by the normalized scaling parameter, and the multiplication result is added to the normalized offset parameter to generate dynamic filter parameters. The dynamic filter parameters are then organized according to the filter space arrangement information, the input channel arrangement information, and the output channel arrangement information to generate a dynamic filter bank.

[0115] In one embodiment, note that the modulation module 40 is specifically used for: Obtain query projection parameters, key projection parameters, and value projection parameters; determine interval sampling positions from the feature map to be upsampled; extract channel features from each interval sampling position; organize the channel features according to the spatial arrangement of each interval sampling position; and generate a downsampled feature map. Based on the query projection parameters, channel projection is performed on the channel features corresponding to each spatial position in the downsampled feature map to generate query features corresponding to each spatial position. The query features are arranged according to their spatial positions in the downsampled feature map to generate a query feature representation. Based on the key projection parameters, feature projection is performed on the conditional latent representation to generate multiple key features arranged along the conditional feature dimension. The key features are combined to generate a key feature representation. Based on the value projection parameters, feature projection is performed on the conditional latent representation to generate multiple value features arranged along the conditional feature dimension. The value features are combined to generate a value feature representation. For each query feature corresponding to a spatial location in the query feature representation, the query feature is matched with each key feature by a dot product to generate an attention matching representation for each spatial location on the condition feature dimension. The attention matching representation corresponding to each spatial location is normalized along the conditional feature dimension to generate the spatial attention weight corresponding to each spatial location on the conditional feature dimension. According to the spatial attention weights corresponding to each spatial location on the conditional feature dimension, the value features are weighted and aggregated to generate modulation feature representations corresponding to each spatial location. The modulation feature representations are arranged according to their spatial locations in the downsampled feature map to generate attention modulation information.

[0116] In one embodiment, the modulation upsampling module 50 is specifically used for: Acquire filter spatial arrangement information, input channel arrangement information, and output channel arrangement information, and extract the dynamic filter parameter set from the dynamic filter bank; Extract modulation feature representations arranged in spatial position from the attention modulation information, aggregate the modulation feature representations according to the spatial position arrangement order of each modulation feature representation, and generate filter modulation feature representations; Based on the filter spatial arrangement information, the input channel arrangement information, and the output channel arrangement information, the filter modulation feature representation is converted into a filter modulation parameter set. The filter modulation parameter set is superimposed with the dynamic filter parameter set to generate a modulated dynamic filter parameter set. The modulated dynamic filter parameter set is organized according to the filter spatial arrangement information, the input channel arrangement information, and the output channel arrangement information to generate a modulated dynamic filter bank. Determine the spatial location to be processed in the feature map to be upsampled, extract local feature regions from each spatial location to be processed, use the modulated dynamic filter bank to perform convolutional feature mapping on each local feature region, generate channel features to be rearranged corresponding to each spatial location to be processed, arrange each channel feature to be rearranged according to each spatial location to be processed, and generate a feature map to be rearranged. Based on the output channel arrangement information, an upsampling position correspondence is generated between each output channel feature in the channel features to be rearranged and the upsampling spatial position corresponding to each spatial position in the feature map to be rearranged. Based on the upsampling position correspondence, the output channel features in the channel features to be rearranged corresponding to each spatial position in the feature map to be rearranged are rearranged to the corresponding upsampling spatial position to generate an upsampling feature map.

[0117] In one embodiment, the image output module 60 is specifically used for: The upsampled feature map is used as the final upsampled feature map received in the final upsampled stage. The final global feature statistics are extracted from the final upsampled feature map. The conditional latent representation and the final global feature statistics are combined into a final filter to generate a feature representation. In the final-stage upsampling stage, a final-stage dynamic filter bank is generated based on the feature representation generated by the final-stage filter, the final-stage feature map to be upsampled is downsampled to generate a final-stage downsampled feature map, and final-stage attention modulation information is generated based on the final-stage downsampled feature map and the conditional latent representation. In the final-stage upsampling stage, the final-stage dynamic filter bank is modulated based on the final-stage attention modulation information to generate a final-stage modulated dynamic filter bank. The final-stage upsampled feature map is then upsampled based on the final-stage modulated dynamic filter bank to generate a final-stage upsampled feature map. The final-level upsampled feature map is passed to the output layer, which includes image channel mapping parameters. Based on the image channel mapping parameters, the final-level upsampled feature map is mapped to image channels to generate an image channel feature map. The generated image is obtained by arranging the feature values ​​of each channel according to their spatial position in the image channel feature map and the image channels.

[0118] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When executed by the processor, the computer program implements the functions or steps of a server-side image generation method based on a dynamic filter bank.

[0119] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements client-side functions or steps of an image generation method based on a dynamic filter bank.

[0120] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the generated input information and the conditional input information, convert the generated input information into an initial feature map, and encode the conditional input information into a conditional latent representation; The initial feature map is transformed to generate a feature map to be upsampled. Global feature statistics are extracted from the feature map to be upsampled. The conditional latent representation and the global feature statistics are combined into a filter to generate a feature representation. A dynamic filter bank is generated based on the filter generation feature representation; The upsampled feature map is downsampled to generate a downsampled feature map, and attention modulation information is generated based on the downsampled feature map and the conditional latent representation. Based on the attention modulation information, the dynamic filter bank is modulated to generate a modulated dynamic filter bank. Based on the modulated dynamic filter bank, the feature map to be upsampled is upsampled to generate an upsampled feature map. The upsampled feature map is converted into a generated image.

[0121] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, and a computer program is stored thereon, which, when executed by a processor, performs the following steps: Obtain the generated input information and the conditional input information, convert the generated input information into an initial feature map, and encode the conditional input information into a conditional latent representation; The initial feature map is transformed to generate a feature map to be upsampled. Global feature statistics are extracted from the feature map to be upsampled. The conditional latent representation and the global feature statistics are combined into a filter to generate a feature representation. A dynamic filter bank is generated based on the filter generation feature representation; The upsampled feature map is downsampled to generate a downsampled feature map, and attention modulation information is generated based on the downsampled feature map and the conditional latent representation. Based on the attention modulation information, the dynamic filter bank is modulated to generate a modulated dynamic filter bank. Based on the modulated dynamic filter bank, the feature map to be upsampled is upsampled to generate an upsampled feature map. The upsampled feature map is converted into a generated image.

[0122] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0123] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0124] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0125] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0126] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An image generation method based on a dynamic filter bank, characterized in that, Includes the following steps: Obtain the generated input information and the conditional input information, convert the generated input information into an initial feature map, and encode the conditional input information into a conditional latent representation; The initial feature map is transformed to generate a feature map to be upsampled. Global feature statistics are extracted from the feature map to be upsampled. The conditional latent representation and the global feature statistics are combined into a filter to generate a feature representation. A dynamic filter bank is generated based on the filter generation feature representation; The upsampled feature map is downsampled to generate a downsampled feature map, and attention modulation information is generated based on the downsampled feature map and the conditional latent representation. Based on the attention modulation information, the dynamic filter bank is modulated to generate a modulated dynamic filter bank. Based on the modulated dynamic filter bank, the feature map to be upsampled is upsampled to generate an upsampled feature map. The upsampled feature map is converted into a generated image.

2. The image generation method based on dynamic filter banks as described in claim 1, characterized in that, Acquiring generated input information and conditional input information, converting the generated input information into an initial feature map, and encoding the conditional input information into a conditional latent representation, including: Obtain the generated input information and conditional input information, input the generated input information into the input layer of the generation network, and use the input layer to map the generated input information into an initial feature representation; Arrange the initial feature representations according to their spatial location and feature channels to generate an initial feature map; The conditional input information is parsed to generate a sequence of conditional data units arranged in the input order; The conditional data unit sequence is input into a conditional coding network, which includes a set of conditional embedding parameters and multiple attention coding units connected sequentially along the feature propagation direction. The conditional embedding vectors corresponding to each conditional data unit in the conditional data unit sequence are extracted based on the set of conditional embedding parameters. The conditional embedding vectors are combined according to the order of the conditional data unit sequence to generate a conditional embedding sequence. The conditional embedding sequence is input into the attention encoding unit located at the beginning of the feature propagation direction, and the conditional embedding sequence is feature-encoded using the attention encoding unit located at the beginning of the feature propagation direction to generate conditional encoded features. Along the feature transmission direction, the conditional coding features generated by the previous attention coding unit are input into the next attention coding unit for feature encoding, and the conditional coding features generated by the attention coding unit located at the end of the feature transmission direction are determined as conditional sequence features. The conditional sequence features are aggregated along the arrangement direction of the conditional data unit sequence to generate a conditional latent representation.

3. The image generation method based on dynamic filter banks as described in claim 1, characterized in that, The initial feature map is subjected to feature transformation to generate a feature map to be upsampled. Global feature statistics are extracted from the feature map to be upsampled. The conditional latent representation and the global feature statistics are combined into a filter to generate a feature representation, including: The initial feature map is input into a feature transformation network, which includes multiple residual feature transformation units connected sequentially along the feature transformation propagation direction. The initial feature map is determined as the input feature map of the residual feature transformation unit located at the beginning of the feature transformation propagation direction. The input feature map is convolved and mapped using the residual feature transformation unit located at the beginning of the feature transformation propagation direction to generate a mapped feature map. The input feature map and the mapped feature map are superimposed according to their corresponding spatial positions and corresponding feature channels to generate an output feature map. Along the direction of feature transformation propagation, the output feature map generated by the previous residual feature transformation unit is determined as the input feature map of the next residual feature transformation unit. The input feature map is then convolved with the next residual feature transformation unit to generate a mapped feature map. The input feature map and the mapped feature map are then superimposed according to their corresponding spatial positions and corresponding feature channels to generate the output feature map. The output feature map generated by the residual feature transformation unit located at the end of the feature transformation transmission direction is determined as the feature map to be upsampled; The feature values ​​of each spatial location corresponding to each feature channel in the feature map to be upsampled are averaged and aggregated to generate channel statistical feature values ​​corresponding to each feature channel. The channel statistical feature values ​​are arranged according to the order of feature channel arrangement to generate global feature statistics. The conditional latent representation and the global feature statistics are concatenated along the feature dimension to generate a feature representation using a filter.

4. The image generation method based on dynamic filter banks as described in claim 1, characterized in that, Generate a dynamic filter bank based on the filter generation feature representation, including: Obtain filter configuration data, normalized scaling parameters, and normalized offset parameters. The filter configuration data includes filter spatial arrangement information, input channel arrangement information, and output channel arrangement information. The filter generation feature representation is input to the filter generation network, which includes a plurality of parameter mapping units connected sequentially along the parameter mapping transfer direction. The parameter mapping unit located at the beginning of the parameter mapping transmission direction is used to perform parameter mapping and nonlinear activation on the filter generation feature representation to generate parameter mapping features; Along the parameter mapping transmission direction, the parameter mapping features generated by the previous parameter mapping unit are input into the next parameter mapping unit for parameter mapping and nonlinear activation. The parameter mapping features generated by the parameter mapping unit located at the end of the parameter mapping transmission direction are determined as the filter parameter representation. Based on the filter space arrangement information, the input channel arrangement information, and the output channel arrangement information, the filter parameter representation is rearranged and reorganized to generate a candidate dynamic filter bank; Extract the parameters of each candidate filter in the candidate dynamic filter bank, square each candidate filter parameter, accumulate and take the square root of each squared candidate filter parameter to generate the filter normalization quantity. Using the filter normalization factor as the divisor, the parameters of each candidate filter are divided to generate the normalized filter parameters corresponding to each candidate filter parameter. Each normalized filter parameter is multiplied by the normalized scaling parameter, and the multiplication result is added to the normalized offset parameter to generate dynamic filter parameters. The dynamic filter parameters are then organized according to the filter space arrangement information, the input channel arrangement information, and the output channel arrangement information to generate a dynamic filter bank.

5. The image generation method based on dynamic filter banks as described in claim 1, characterized in that, The upsampled feature map is downsampled to generate a downsampled feature map. Attention modulation information is generated based on the downsampled feature map and the conditional latent representation, including: Obtain query projection parameters, key projection parameters, and value projection parameters; determine interval sampling positions from the feature map to be upsampled; extract channel features from each interval sampling position; organize the channel features according to the spatial arrangement of each interval sampling position; and generate a downsampled feature map. Based on the query projection parameters, channel projection is performed on the channel features corresponding to each spatial position in the downsampled feature map to generate query features corresponding to each spatial position. The query features are arranged according to their spatial positions in the downsampled feature map to generate a query feature representation. Based on the key projection parameters, feature projection is performed on the conditional latent representation to generate multiple key features arranged along the conditional feature dimension. The key features are combined to generate a key feature representation. Based on the value projection parameters, feature projection is performed on the conditional latent representation to generate multiple value features arranged along the conditional feature dimension. The value features are combined to generate a value feature representation. For each query feature corresponding to a spatial location in the query feature representation, the query feature is matched with each key feature by a dot product to generate an attention matching representation for each spatial location on the condition feature dimension. The attention matching representation corresponding to each spatial location is normalized along the conditional feature dimension to generate the spatial attention weight corresponding to each spatial location on the conditional feature dimension. According to the spatial attention weights corresponding to each spatial location on the conditional feature dimension, the value features are weighted and aggregated to generate modulation feature representations corresponding to each spatial location. The modulation feature representations are arranged according to their spatial locations in the downsampled feature map to generate attention modulation information.

6. The image generation method based on dynamic filter banks as described in claim 1, characterized in that, The dynamic filter bank is modulated based on the attention modulation information to generate a modulated dynamic filter bank. The feature map to be upsampled is then upsampled based on the modulated dynamic filter bank to generate an upsampled feature map, including: Acquire filter spatial arrangement information, input channel arrangement information, and output channel arrangement information, and extract the dynamic filter parameter set from the dynamic filter bank; Extract modulation feature representations arranged in spatial position from the attention modulation information, aggregate the modulation feature representations according to the spatial position arrangement order of each modulation feature representation, and generate filter modulation feature representations; Based on the filter spatial arrangement information, the input channel arrangement information, and the output channel arrangement information, the filter modulation feature representation is converted into a filter modulation parameter set. The filter modulation parameter set is superimposed with the dynamic filter parameter set to generate a modulated dynamic filter parameter set. The modulated dynamic filter parameter set is organized according to the filter spatial arrangement information, the input channel arrangement information, and the output channel arrangement information to generate a modulated dynamic filter bank. Determine the spatial location to be processed in the feature map to be upsampled, extract local feature regions from each spatial location to be processed, use the modulated dynamic filter bank to perform convolutional feature mapping on each local feature region, generate channel features to be rearranged corresponding to each spatial location to be processed, arrange each channel feature to be rearranged according to each spatial location to be processed, and generate a feature map to be rearranged. Based on the output channel arrangement information, an upsampling position correspondence is generated between each output channel feature in the channel features to be rearranged and the upsampling spatial position corresponding to each spatial position in the feature map to be rearranged. Based on the upsampling position correspondence, the output channel features in the channel features to be rearranged corresponding to each spatial position in the feature map to be rearranged are rearranged to the corresponding upsampling spatial position to generate an upsampling feature map.

7. The image generation method based on dynamic filter banks as described in claim 1, characterized in that, Converting the upsampled feature map into a generated image includes: The upsampled feature map is used as the final upsampled feature map received in the final upsampled stage. The final global feature statistics are extracted from the final upsampled feature map. The conditional latent representation and the final global feature statistics are combined into a final filter to generate a feature representation. In the final-stage upsampling stage, a final-stage dynamic filter bank is generated based on the feature representation generated by the final-stage filter, the final-stage feature map to be upsampled is downsampled to generate a final-stage downsampled feature map, and final-stage attention modulation information is generated based on the final-stage downsampled feature map and the conditional latent representation. In the final-stage upsampling stage, the final-stage dynamic filter bank is modulated based on the final-stage attention modulation information to generate a final-stage modulated dynamic filter bank. The final-stage upsampled feature map is then upsampled based on the final-stage modulated dynamic filter bank to generate a final-stage upsampled feature map. The final-level upsampled feature map is passed to the output layer, which includes image channel mapping parameters. Based on the image channel mapping parameters, the final-level upsampled feature map is mapped to image channels to generate an image channel feature map. The generated image is obtained by arranging the feature values ​​of each channel according to their spatial position in the image channel feature map and the image channels.

8. An image generation device based on a dynamic filter bank, characterized in that, The image generation device based on dynamic filter banks includes: The input representation module is used to acquire generated input information and conditional input information, convert the generated input information into an initial feature map, and encode the conditional input information into a conditional latent representation; The filtering feature module is used to perform feature transformation on the initial feature map to generate a feature map to be upsampled, extract global feature statistics from the feature map to be upsampled, and combine the conditional latent representation and the global feature statistics into a filter to generate a feature representation. A filter generation module is used to generate a dynamic filter bank based on the filter generation feature representation. Attention modulation module is used to downsample the feature map to be upsampled to generate a downsampled feature map, and generate attention modulation information based on the downsampled feature map and the conditional latent representation; The modulation upsampling module is used to modulate the dynamic filter bank based on the attention modulation information to generate a modulated dynamic filter bank, and to upsample the feature map to be upsampled based on the modulated dynamic filter bank to generate an upsampled feature map. An image output module is used to convert the upsampled feature map into a generated image.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and an image generation program based on a dynamic filter bank stored in the memory and executable on the processor. When executed by the processor, the image generation program based on a dynamic filter bank implements the steps of the image generation method based on a dynamic filter bank as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores an image generation program based on a dynamic filter bank, which, when executed by a processor, implements the steps of the image generation method based on a dynamic filter bank as described in any one of claims 1-7.

Images