Mixed linear layer enhanced image generation method and system

Abstract

The invention provides a mixed linear layer enhanced image generation method and system, and the method comprises the steps: a visual information processing step: carrying out the marking processing of the obtained visual information, extracting the original features of the visual information, and outputting a noise enhanced residual visual mark; a text information processing step: obtaining weight parameters of an INR encoder through a super network; a color feature acquisition step: performing mapping processing on the coordinate information of the visual information by using a mixed linear layer and an activation function of an INR encoder, and acquiring a color feature value of a corresponding space coordinate; and an image synthesis step: fusing and inputting the color features generated in the color feature acquisition step and the noise residual error marks obtained in the visual information processing step into a multi-layer perceptron network, and synthesizing an image with high continuity. The method supports any resolution output, breaks through the limitation of traditional pixel representation, and improves the expression capability and adaptability of the model.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to an image generation method and system with hybrid linear layer enhancement. Background Technology

[0002] Humans are born with the ability to correlate linguistic descriptions with visual imagination, an ability that has inspired the development of the "Text-to-Image Generation" (T2I) task in the field of artificial intelligence. This task aims to automatically generate high-quality images that are semantically consistent with the descriptions given natural language input. Compared to other forms of conditional input (such as sketches, labels, or semantic graphs), text descriptions offer greater expressive flexibility and intuitiveness, allowing users to express their ideas about the content of a target image more naturally.

[0003] Traditional text-to-image generation methods primarily rely on deep generative models, such as Generative Adversarial Networks (GANs) and diffusion models. These methods typically represent images as discrete two-dimensional pixel matrices. While this representation has made significant progress in image modeling, it has limitations in terms of resolution, scale adaptability, and generative continuity. For example, when increasing image resolution, discrete pixel-based generative models often require a simultaneous increase in the model's parameter size, leading to increased training difficulty and a surge in computational resource consumption. This is especially true in high-resolution scenarios, where batch processing size is limited by GPU memory capacity, resulting in a significant decrease in training efficiency.

[0004] In recent years, implicit neural representation (INR) has provided a novel perspective for image modeling. This representation not only avoids the quantization limitations of discrete pixels but also naturally supports image generation at arbitrary resolutions and exhibits good stability in terms of memory usage. Furthermore, INR possesses advantages such as image boundary extrapolation, accelerated inference for low-resolution images, and plug-and-play super-resolution capabilities, bringing new possibilities to image generation tasks.

[0005] Current mainstream T2I generation methods typically use text embeddings to adjust the hidden states of the generator or discriminator to achieve semantic understanding of the text and control over the image content. However, this embedding-based conditional generation approach still has limitations in terms of expressive power and semantic alignment. In recent years, HyperNetwork technology has been introduced into conditional generation tasks. Its core idea is to dynamically generate the weight parameters of the main network based on input conditions (such as text descriptions), thereby generating a customized model instance for each conditional input. Research shows that this approach has stronger expressive power and generation flexibility compared to traditional embedded conditional adjustment methods. However, existing technologies still have the following problems: (1) Image representation limits resolution and generalization ability: Current mainstream text-to-image generation methods usually represent images as discrete two-dimensional pixel matrices. When improving image resolution, this representation often requires a simultaneous increase in the model parameter scale, resulting in high computational resource consumption, low training efficiency, and difficulty in achieving arbitrary resolution output and high-quality reconstruction of images.

[0006] (2) Limited expressive power of conditional generation mechanisms: Most existing methods use text embedding to adjust the hidden state of the generator or discriminator in order to understand the semantics of the text and control the image content. However, this embedding-based conditional adjustment method has insufficient expressive power in terms of semantic alignment and generation control, and it is difficult to generate images with rich details that are highly consistent with the text description. Summary of the Invention

[0007] To address the problems in the prior art, this invention provides a hybrid linear layer enhanced image generation method and system that enables models to generate images at arbitrary resolutions, no longer limited by fixed resolution pixel grids, and generates more refined images that are highly consistent with text descriptions.

[0008] The image generation method of the present invention with hybrid linear layer enhancement includes the following steps: Visual information processing steps: The acquired visual information is labeled, the original features of the visual information are extracted, noise is superimposed on the original features, and noise-enhanced residual visual labels are output. Text information processing steps: Encode the input text information and obtain the weight parameters of the INR encoder through a hypernetwork. INR is a hidden neural representation. Color feature acquisition steps: Based on the weight parameters, the coordinate information of visual information is mapped using the hybrid linear layer and activation function of the INR encoder to obtain the color feature value of the corresponding spatial coordinates; Image synthesis step: The color features generated in the color feature acquisition step are fused with the noise residual labels obtained in the visual information processing step and input into the multilayer perceptron network to synthesize an image with high continuity.

[0009] Furthermore, in the visual information processing step, the visual information is one or more image information. After acquiring the image information, it is preprocessed to convert it into a digital matrix structure that a computer can process. Each element in the digital matrix structure represents the color value of a pixel. The converted image information... Represented as: Where H and W are image information respectively. Height and width.

[0010] Furthermore, after preprocessing the visual information, the method for tokenizing the image information is as follows: features are extracted from the image using convolutional neural networks (CNNs). CNNs include multiple levels of convolutional layers and pooling layers. Output after layer convolution operation Represented as: in, This represents the convolution operation. and They are the first The weight matrix and bias vector of the layer, It is an activation function. Through multiple convolutional and pooling operations, higher-level original features Z are gradually obtained: in, These are the height, width, and number of channels of the new feature map. After the tokenization process, the original feature Z is added to the mean. =0, variance Gaussian noise The results of noise-enhanced residual visual labeling Expressed as a formula: .

[0011] Furthermore, in the text information processing step, the method for encoding the input text information is as follows: (1) The text information is encoded using a pre-trained language model and converted into a continuous vector. The conversion formula is as follows: in, Represents the text embedding matrix. It is the length of the text sequence. It represents the embedding dimension of each word, where T is time. (2) Introduce random noise that follows a standard normal distribution. , Embed the encoded text into the matrix and random noise The vectors are concatenated to form the input vector of the hypernetwork. .

[0012] Furthermore, the method for obtaining the weight parameters of the INR encoder through the hypernetwork is as follows: Input vector The input vector is fed into a hypernetwork, which processes it. The hypernetwork has a multi-layer structure, and the processing formula for each layer is as follows: Where K is the number of layers, This represents the activation function. W K and b K These are the weight matrix and bias vector of the Kth layer, respectively. For the output of each layer, For inputs from the second layer and above, After multiple processing steps, the final output is the weight parameters of the INR encoder. The formula for calculating the weight parameters is as follows: The weight parameters The implicit neural representation (INR) model is used to construct the image generation network for subsequent image generation tasks. Based on this weight parameter, textual information is dynamically mapped into the structure of the image generation network.

[0013] Furthermore, in the color feature acquisition step, the specific processing method for acquiring color feature values ​​is as follows: (1) Image coordinate information , N Given the number of pixels, an initial feature transformation is performed using a linear mapping layer that blends linear layers: in, and These are the weight matrix and bias vector parameters of the linear mapping layer. It is a preliminary feature representation of coordinate information; (2) Mix the preliminary features of coordinate information with text information to obtain the mixed attention features. : in, This represents the Hadamard product, which is the product of elements. It is the first Word vectors, and These are the weight matrix and bias vector parameters of the attention module. Through feature fusion, the model can dynamically focus on regions in the image related to the current word based on the text description. (3) All attention features are weighted and summed, and the final implicit neural representation color feature value R is obtained through the final linear mapping layer: in, and These are the weight matrix and bias vector parameters of the final linear layer, respectively. The implicit neural representation is a continuous function that can output the corresponding RGB color value for any coordinate, thereby realizing image reconstruction and generation at any resolution. (4) The INR encoder outputs the color feature value R corresponding to the spatial coordinates.

[0014] Furthermore, sub-step (2) for obtaining color feature values ​​also includes a super-attention processing step for the text information. The processing method for the super-attention processing step is as follows: The original weight matrix After low-rank decomposition or X-factor decomposition, the outer product of two or more low-dimensional vectors is obtained. A word-based super-attention module calculates the importance weight of each word in the text information, selects the most relevant word vectors for fusion, and obtains the semantic importance of each word in the input text. Then, the corresponding low-rank hybrid weights are dynamically selected and fused to generate the attention hybrid weights. This causes the model to focus on the words most relevant to the current image region, using a mixed attention weighting. The calculation formula is: in, It is the initial weight matrix of the current layer. It is a set of word-level mixed tensors. for The transpose of the matrix, It normalizes word similarity using a sentence-level hybrid tensor, generating attention weights. It is a weighted fusion of word-level hybrid tensors. The number of input features, The number of output features, The final weight matrix of the hybrid processing It consists of two parts, the first part being the weights. , derived from sentence-level hybrid tensors, through noise vectors zand text embedding matrix Generation; the second part is the attention-mixed weights. , a word-level hybrid tensor generated from the WHAtt mechanism.

[0015] The present invention also provides an INR-GAN image generation system with hybrid linear layer enhancement, used to implement the aforementioned image generation method with hybrid linear layer enhancement, the INR-GAN image generation system with hybrid linear layer enhancement comprising: Visual information processing module: used to perform tagging processing on the acquired visual information, extract the original features of the visual information, superimpose noise on the original features, and output noise-enhanced residual visual tags; Text information processing module: used to encode the input text information and obtain the weight parameters of the INR encoder through the hypernetwork. INR is a hidden neural representation. Color feature acquisition module: Used to map the coordinate information of visual information using the hybrid linear layer and activation function of the INR encoder to obtain the color feature value of the corresponding spatial coordinates; Image synthesis module: It is used to fuse the color features generated in the color feature acquisition step with the noise residual labels obtained in the visual information processing step and input them into the multilayer perceptron network to synthesize an image with high continuity.

[0016] Furthermore, the visual information processing module is implemented using deep learning convolutional neural networks (CNNs), and the text information processing module includes a text information encoding submodule and a weight parameter acquisition submodule. The text information encoding submodule uses a hypernetwork structure to acquire weight parameters, which are used to construct an implicit neural representation (INR) model to perform subsequent image generation tasks. Based on these weight parameters, text information is dynamically mapped into the structure of the image generation network. The color feature acquisition step is implemented using an INR encoder. A hybrid linear layer is introduced into the backbone network of the INR encoder to inject text conditional information into the image generation process in a parameter mixing manner.

[0017] Furthermore, in the color feature acquisition module, an implicit neural representation (INR) modeling method is used to express the image as a continuous spatial function, enabling the model to generate images at arbitrary resolutions. The color feature acquisition module includes the following sub-steps: (1) Initial feature transformation unit of image: used to perform initial feature transformation on the coordinate information of image through the linear mapping layer of hybrid linear layer to obtain the preliminary features of the transformed coordinate information; (2) Feature fusion unit: used to blend the preliminary features of coordinate information with text information to obtain the blended attention features; (3) Color feature value acquisition unit: used to sum all attention features by weight and obtain the color feature value of the final implicit neural representation through the final linear mapping layer; (4) Output unit: used to output the color feature value corresponding to the spatial coordinates.

[0018] Compared with the prior art, the beneficial effects of the present invention are: (1) Supports arbitrary resolution output, breaking through the limitations of traditional pixel representation. This invention adopts the implicit neural representation (INR) modeling method to express the image as a continuous spatial function, enabling the model to generate images at arbitrary resolutions, no longer limited by a fixed resolution pixel grid. It is particularly suitable for applications such as high-resolution image generation, image super-resolution, and continuous scale output.

[0019] (2) Enhancing model expressiveness and enabling personalized model customization. This invention introduces a hypernetwork structure to dynamically generate the weight parameters of the main network based on the input text description, generating a customized model instance for each text input. Compared with models with fixed parameters, this method significantly improves the model's expressiveness and adaptability, enabling more accurate matching of text semantics and generating image content that conforms to the description.

[0020] (3) Enhance the semantic alignment between images and text. By introducing a hybrid linear layer into the backbone network of the INR encoder, textual conditional information is injected into the image generation process in a parameter-mixing manner. This mechanism can fine-grainedly adjust the weights of each layer in the network, thereby controlling the local details and overall structure of the image more precisely, so that the generated image is highly consistent with the input text in semantics.

[0021] (4) Enhanced controllability and interpretability of image generation. The text-driven dynamic parameter generation mechanism of this invention makes the image generation process more interpretable and controllable. By adjusting the input text, users can directly influence the content, style, and structure of the generated image, improving the flexibility and practicality of human-computer interaction. Attached Figure Description

[0022] To more clearly illustrate the solutions in this invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart of the image generation method with hybrid linear layer enhancement according to the present invention; Figure 2 This is a system structure block diagram of the image generation method with hybrid linear layer enhancement according to the present invention. Detailed Implementation

[0024] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings are used to distinguish different objects, not to describe a particular order.

[0025] In this invention, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment to other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this invention can be combined with other embodiments.

[0026] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0027] like Figure 1 As shown, the image generation method with hybrid linear layer enhancement of the present invention includes the following steps: Visual information processing step S1: The acquired visual information is labeled, the original features of the visual information are extracted, noise is superimposed on the original features, and noise-enhanced residual visual labels are output. Text information processing step S2: Encode the input text information and obtain the weight parameters of the INR encoder through the hypernetwork. INR is a hidden neural representation. Color feature acquisition step S3: Based on the weight parameters, the coordinate information of the visual information is mapped using the hybrid linear layer and activation function of the INR encoder to obtain the color feature value of the corresponding spatial coordinates. Image synthesis step S4: The color features generated in the color feature acquisition step are fused with the noise residual labels obtained in the visual information processing step and input into the multilayer perceptron network to synthesize an image with high continuity.

[0028] This invention models images as continuous functions using the INR (Integrated Linear Representation), enabling the model to generate images at arbitrary resolutions, overcoming the resolution limitations of traditional discrete pixel representations. Building upon this, a hypernetwork structure is introduced to dynamically generate the weight parameters of the main network based on the input text description, thus generating a customized model instance for each text input, significantly enhancing the model's expressive power and generation control precision. Furthermore, a hybrid linear layer is added to the INR backbone network, injecting textual information as a generation condition into the image generation process through parameter mixing. This approach allows for finer control over the local details and overall structure of the image, thereby improving the semantic alignment between the generated image and the text description.

[0029] The method of the present invention will be described in detail below through specific algorithm embodiments.

[0030] Step S1: Visual information processing.

[0031] In this step, the input visual information is typically one or more images. To enable the computer to process these images, they first need to be converted into a digital matrix form, where each element represents the color value of a pixel. After acquiring the image information, it undergoes preprocessing to convert it into a digital matrix structure that the computer can process. Each element in the digital matrix structure represents the color value of a pixel. The converted image information... Represented as: Where H and W are image information respectively. Height and width.

[0032] After visual information preprocessing, the next step is to tokenize these digitally represented images. In deep learning, this process is typically accomplished by convolutional neural networks (CNNs), which extract features from the images through a series of convolutional and pooling layers.

[0033] The method for tokenizing image information according to this invention is as follows: Convolutional Neural Networks (CNNs) extract features from images. CNNs consist of multiple levels of convolutional layers and pooling layers. Output after layer convolution operation Represented as: in, This represents the convolution operation. and They are the first The weight matrix and bias vector of the layer, It is an activation function. Through multiple convolutional and pooling operations, higher-level original features Z are gradually obtained: in, These are the height, width, and number of channels of the new feature map. After obtaining the high-level feature representation of the image, to enhance the model's generalization ability and simulate uncertainties in real-world applications, this invention superimposes Gaussian noise onto these features. This invention adds the original feature Z to the mean. =0, variance Gaussian noise The results of noise-enhanced residual visual labeling Expressed as a formula: .

[0034] By introducing noise (such as Gaussian noise) into the feature representation, the model can better cope with subtle variations and uncertainties in the input data, improving not only its generalization ability but also its robustness. These noise-enhanced visual labels are then applied to various tasks, including but not limited to computer vision tasks such as classification, object detection, and image generation. Noise enhancement can effectively increase the model's tolerance to data variability during the training phase, thereby improving its performance and reliability in real-world applications.

[0035] Step S2: Text information processing.

[0036] The method for encoding the input text information is as follows: (1) The text information is encoded using a pre-trained language model and converted into a continuous vector. The conversion formula is as follows: in, Represents the text embedding matrix. It is the length of the text sequence. T represents the embedding dimension of each word, and T represents time. This encoding method not only preserves the semantic information of the text, but also provides a structured input foundation for subsequent model processing.

[0037] (2) In order to enhance the robustness and generalization ability of the model, avoid overfitting of the model to the training data, and simulate the uncertainties that may be encountered in actual applications, random noise following a standard normal distribution is introduced. , The introduction of this noise enables the model to better adapt to small perturbations when faced with different inputs, thereby improving its stability and performance in practical applications.

[0038] Embed the encoded text into the matrix and random noise The vectors are concatenated to form the input vector of the hypernetwork. This concatenation operation allows textual and noise information to work synergistically in subsequent processing, preserving the semantic guidance of the text while enhancing the diversity of the model. Next, the input is fed into the supernetwork.

[0039] The method for obtaining the weight parameters of the INR encoder through a hypernetwork in this invention is as follows: Input vector The input vector is fed into a hypernetwork, which processes it. The hypernetwork has a multi-layer structure, and the processing formula for each layer is as follows: Where K is the number of layers, This represents the activation function. W K and b K These are the weight matrix and bias vector of the Kth layer, respectively. For the output of each layer, For inputs from the second layer and above, After multiple processing steps, the final output is the weight parameters of the INR encoder. The formula for calculating the weight parameters is as follows: The weight parameters The implicit neural representation (INR) model is used to construct the image generation network for subsequent image generation tasks. Based on this weight parameter, textual information is dynamically mapped into the structure of the image generation network.

[0040] The hypernetwork structure in this example has four layers. The first layer of the hypernetwork uses a fully connected layer and an activation function to process the input. After preliminary processing, the output value of the first layer is obtained. : 1 and b 1 represents the weight matrix and bias vector of the first layer, which performs a non-linear transformation on the input and extracts low-level feature combinations.

[0041] The second layer then continues to process the output of the previous layer: This layer further extracts higher-level features and enhances the model's ability to learn complex patterns.

[0042] The third layer further abstracts the feature representation: Through multi-layer nonlinear transformations, the supernetwork can generate highly nonlinear and expressive feature representations, providing a foundation for subsequent INR encoder weight generation.

[0043] Finally, the supernetwork outputs the weight parameters of the INR encoder: in , These are the weight matrix and bias vector of the output layer. These weight parameters will be used to construct the implicit neural representation (INR) model for subsequent image generation tasks. In this way, textual information is dynamically mapped into the structure of the image generation network, achieving an accurate response to textual descriptions. Step S3: Color feature acquisition.

[0044] While processing the text information, the coordinate information of the image is also processed. The specific processing method is as follows: (1) Image coordinate information , N Given the number of pixels, an initial feature transformation is performed using a linear mapping layer that blends linear layers: in, and These are the weight matrix and bias vector parameters of the linear mapping layer. This is a preliminary feature representation of coordinate information, and this step ensures that spatial information can be effectively integrated with textual information.

[0045] (2) Mix the preliminary features of coordinate information with text information to obtain the mixed attention features. : in, This represents the Hadamard product, which is the product of elements. It is the first Word vectors, and These are the weight matrix and bias vector parameters of the attention module. Through feature fusion, the model can dynamically focus on regions in the image related to the current word based on the text description. This mechanism enables the model to dynamically focus on regions in the image related to the current word based on the text description, thereby improving the semantic consistency of the generated image.

[0046] (3) All attention features are weighted and summed, and the final implicit neural representation color feature value R is obtained through the final linear mapping layer: in, and These are the weight matrix and bias vector parameters of the final linear layer, respectively. The implicit neuron is represented as a continuous function that can adapt to any coordinates ( x , y It outputs the corresponding RGB color values, thereby enabling image reconstruction and generation at any resolution.

[0047] (4) The INR encoder outputs the color feature value R corresponding to the spatial coordinates.

[0048] Furthermore, in the sub-step (2) of obtaining color feature values, in order to enhance the semantic alignment between text and coordinates, an attention mechanism is introduced to process each word vector in the text. vi The input features are then fused with coordinate features. This example also includes a super-attention processing step for the text information, which is mainly achieved through a hybrid linear layer integrated into the INR encoder. In this example, the hybrid linear layer is one of the core components of the INR encoder. It combines linear mapping, Hadamard product, and word-based super-attention mechanism to achieve effective mixing and enhancement of input features.

[0049] The processing method for the super-attention processing step described in this example is as follows: Linear mapping is a fundamental component of hybrid linear layers, used to linearly transform input features. Assume the input features are... Weight matrix Bias vector , The number of input features, Given the number of output features, then the output features after linear transformation The calculation formula is: To further reduce the number of parameters and computational complexity, the hybrid linear layer employs low-rank decomposition and X-factor decomposition techniques. Low-rank decomposition decomposes the high-dimensional weight matrix into the outer product of multiple low-dimensional vectors, while X-factor decomposition is an extreme form of low-rank decomposition, decomposing the weight matrix into the outer product of two low-dimensional vectors.

[0050] This example uses X-factor decomposition as an illustration. Assume the original weight matrix is... After X-factor decomposition, it can be expressed as: in, and They are two low-dimensional vectors.

[0051] The word-based super-attention module calculates the importance weight of each word and selects the most relevant word vectors for fusion. Based on the semantic importance of each word in the input text, it dynamically selects and fuses the corresponding low-rank mixed vectors, thereby achieving refinement of the linear layer in the INR decoder and semantically aware weight mixing. It combines word-level text embedding, low-rank parameter generation, and a learnable attention mechanism, enabling the model to focus on the words most relevant to the current image region (such as "red," "wings," and "background"), improving the semantic consistency and detail quality of the generated image.

[0052] This example uses the original weight matrix. After X-factor decomposition, the outer product of two low-dimensional vectors is obtained. The most relevant word vectors are selected and fused to obtain the semantic importance of each word in the input text. Then, the corresponding low-rank hybrid weights are dynamically selected and fused to generate attention hybrid weights. In this example, attention is mixed with weights. The calculation formula is: in, It is the initial weight matrix of the current layer. It is a set of word-level mixed tensors. for The transpose of the matrix, It normalizes word similarity using a sentence-level hybrid tensor, generating attention weights. It is a weighted and fused word-level hybrid tensor.

[0053] As can be seen from the above process, the final weight matrix of the hybrid processing in this example is... It consists of two parts, the first part being the weights. , derived from sentence-level hybrid tensors, through noise vectors z and text embedding matrix Generation; the second part is the attention-mixed weights. , a word-level hybrid tensor generated from the WHAtt mechanism.

[0054] Step S4: Image synthesis.

[0055] This invention combines the color features output in step S3 with the noise-enhanced residual visual labels retained in step S1, and inputs them together into a subsequent MLP (Multilayer Perceptron) network for final image synthesis. This process is a crucial step in constructing high-quality, highly continuous, and realistic images. The main reason is: First, step S3 is primarily responsible for generating matching color features based on the text description. These color features can be understood as the color information corresponding to various spatial locations (such as pixels or coordinate points) in the image, such as color combinations like red, green, and blue, as well as related visual attributes such as texture and lighting. These features are usually dynamically generated based on text semantics and the model's internal attention mechanism, exhibiting strong semantic consistency; that is, they can effectively reflect the object, scene, or style described in the text.

[0056] On the other hand, step S1 introduces noise-enhanced residual visual markers, which are mainly used to preserve the randomness and diversity in the image generation process. Residual visual markers can be understood as a structured "random perturbation" that helps the model introduce subtle variations when generating images, thus avoiding overly simplistic or unrealistic results. By preserving these residual visual markers and appropriately fusing them in subsequent stages, the naturalness and visual richness of the images can be enhanced.

[0057] During the fusion process, the color features generated in step S3 are concatenated or weighted together with the residual visual labeling information retained in step S1 to form a richer feature representation. This fused feature contains two aspects of information: first, color semantic information guided by textual information; and second, noise perturbation information that preserves the diversity of the generation process. Subsequently, this fused feature is input into an MLP network. The MLP network further integrates and maps these features through multiple layers of nonlinear transformations, gradually constructing the final color output at each location in the image.

[0058] In this example, the MLP network acts as an "image synthesizer" in the process. It is responsible not only for mapping input features to specific color values ​​but also for ensuring the spatial continuity and visual consistency of the image. For example, color transitions between adjacent pixels should be smooth, object edges should be sharp, and the overall image should conform to the semantic content of the text description. By fully fusing the color features from step S3 with the residual visual labeling information from step S1, the MLP can generate images that not only conform to the text description but also possess a high degree of realism and natural visual effect.

[0059] like Figure 2 As shown, the present invention also provides an INR-GAN image generation system with hybrid linear layer enhancement, used to implement the aforementioned image generation method with hybrid linear layer enhancement. The INR-GAN image generation system with hybrid linear layer enhancement includes: Visual information processing module: used to perform tagging processing on the acquired visual information, extract the original features of the visual information, superimpose noise on the original features, and output noise-enhanced residual visual tags; Text information processing module: used to encode the input text information and obtain the weight parameters of the INR encoder through the hypernetwork. INR is a hidden neural representation. Color feature acquisition module: Used to map the coordinate information of visual information using the hybrid linear layer and activation function of the INR encoder to obtain the color feature value of the corresponding spatial coordinates; Image synthesis module: It is used to fuse the color features generated in the color feature acquisition step with the noise residual labels obtained in the visual information processing step and input them into the multilayer perceptron network to synthesize an image with high continuity.

[0060] The visual information processing module is implemented using deep learning convolutional neural networks (CNNs). The text information processing module includes a text information encoding submodule and a weight parameter acquisition submodule. The text information encoding submodule uses a hypernetwork structure to acquire weight parameters, which are used to construct an implicit neural representation (INR) model to perform subsequent image generation tasks. Based on these weight parameters, text information is dynamically mapped into the structure of the image generation network. The color feature acquisition step is implemented using an INR encoder. A hybrid linear layer is introduced into the backbone network of the INR encoder to inject text conditional information into the image generation process in a parameter mixing manner.

[0061] Compared with the prior art, the present invention has the following innovative features: (1) Supports output at any resolution, breaking through the limitations of traditional pixel representation.

[0062] This invention employs an implicit neural representation (INR) modeling approach, expressing images as continuous spatial functions. This allows the model to generate images at arbitrary resolutions, no longer limited by fixed-resolution pixel grids. It is particularly suitable for applications such as high-resolution image generation, image super-resolution, and continuous-scale output.

[0063] (2) Enhance the expressive power of the model and realize personalized model customization.

[0064] This invention introduces a hypernetwork structure to dynamically generate the weight parameters of the main network based on the input text description, creating a customized model instance for each text input. Compared to models with fixed parameters, this method significantly improves the model's expressive power and adaptability, enabling more accurate matching of text semantics and generating image content that conforms to the description.

[0065] (3) Improve the semantic alignment between images and text.

[0066] This invention introduces a hybrid linear layer into the backbone network of the INR encoder, injecting textual conditional information into the image generation process through parameter mixing. This mechanism enables fine-grained adjustment of the weights of each layer in the network, thereby allowing for more precise control over the local details and overall structure of the image, ensuring that the generated image maintains a high degree of semantic consistency with the input text.

[0067] (4) Enhance the controllability and interpretability of image generation.

[0068] The text-driven dynamic parameter generation mechanism of this invention enhances the interpretability and controllability of the image generation process. By adjusting the input text, users can directly influence the content, style, and structure of the generated image, improving the flexibility and practicality of human-computer interaction.

[0069] (5) Modular design, which facilitates expansion and optimization.

[0070] This invention modularly integrates the INR encoder, hypernetwork, and hybrid mechanism, with each part having a clearly defined function and structure, facilitating subsequent model optimization, component replacement, and functional expansion, and exhibiting good engineering feasibility and system compatibility.

[0071] The specific embodiments described above are preferred embodiments of the present invention and are not intended to limit the specific scope of the present invention. The scope of the present invention includes, but is not limited to, these specific embodiments. All equivalent changes made in accordance with the present invention are within the protection scope of the present invention.

Claims

1. An image generation method with hybrid linear layer enhancement, characterized in that, Includes the following steps: Visual information processing steps: The acquired visual information is labeled, the original features of the visual information are extracted, noise is superimposed on the original features, and noise-enhanced residual visual labels are output. Text information processing steps: Encode the input text information and obtain the weight parameters of the INR encoder through a hypernetwork. INR is a hidden neural representation. Color feature acquisition steps: Based on the weight parameters, the coordinate information of visual information is mapped using the hybrid linear layer and activation function of the INR encoder to obtain the color feature value of the corresponding spatial coordinates; Image synthesis step: The color features generated in the color feature acquisition step are fused with the noise residual labels obtained in the visual information processing step and input into the multilayer perceptron network to synthesize an image with high continuity.

2. The image generation method with hybrid linear layer enhancement according to claim 1, characterized in that: In the visual information processing step, the visual information is one or more image information. After acquiring the image information, it is preprocessed to convert it into a digital matrix structure that a computer can process. Each element in the digital matrix structure represents the color value of a pixel. The converted image information... Represented as: Where H and W are image information respectively. Height and width.

3. The image generation method with hybrid linear layer enhancement according to claim 2, characterized in that: After visual information preprocessing, the method for tokenizing image information is as follows: features are extracted from the image using convolutional neural networks (CNNs). CNNs include multiple levels of convolutional layers and pooling layers. Output after layer convolution operation Represented as: in, This represents the convolution operation. and They are the first The weight matrix and bias vector of the layer, It is an activation function. Through multiple convolutional and pooling operations, higher-level original features Z are gradually obtained: in, These are the height, width, and number of channels of the new feature map. After the tokenization process, the original feature Z is added to the mean. =0, variance Gaussian noise The results of noise-enhanced residual visual labeling Expressed as a formula: .

4. The image generation method with hybrid linear layer enhancement according to claim 1, characterized in that: In the text information processing steps, the method for encoding the input text information is as follows: (1) The text information is encoded using a pre-trained language model and converted into a continuous vector. The conversion formula is as follows: in, Represents the text embedding matrix. It is the length of the text sequence. It represents the embedding dimension of each word, where T is time. (2) Introduce random noise that follows a standard normal distribution. , Embed the encoded text into the matrix and random noise The vectors are concatenated to form the input vector of the hypernetwork. .

5. The image generation method with hybrid linear layer enhancement according to claim 4, characterized in that: The method for obtaining the weight parameters of the INR encoder through a hypernetwork is as follows: Input vector The input vector is fed into a hypernetwork, which processes it. The hypernetwork has a multi-layer structure, and the processing formula for each layer is as follows: Where K is the number of layers, This represents the activation function. W K and b K These are the weight matrix and bias vector of the Kth layer, respectively. For the output of each layer, For inputs from the second layer and above, After multiple processing steps, the final output is the weight parameters of the INR encoder. The formula for calculating the weight parameters is as follows: The weight parameters The implicit neural representation (INR) model is used to construct the image generation network for subsequent image generation tasks. Based on this weight parameter, textual information is dynamically mapped into the structure of the image generation network.

6. The image generation method with hybrid linear layer enhancement according to claim 5, characterized in that: In the color feature acquisition step, the specific processing method for obtaining color feature values ​​is as follows: (1) Image coordinate information , N Given the number of pixels, an initial feature transformation is performed using a linear mapping layer that blends linear layers: in, and These are the weight matrix and bias vector parameters of the linear mapping layer. It is a preliminary feature representation of coordinate information; (2) Mix the preliminary features of coordinate information with text information to obtain the mixed attention features. : in, This represents the Hadamard product, which is the product of elements. It is the first Word vectors, and These are the weight matrix and bias vector parameters of the attention module. Through feature fusion, the model can dynamically focus on regions in the image related to the current word based on the text description. (3) All attention features are weighted and summed, and the final implicit neural representation color feature value R is obtained through the final linear mapping layer: in, and These are the weight matrix and bias vector parameters of the final linear layer, respectively. The implicit neural representation is a continuous function that can output the corresponding RGB color value for any coordinate, thereby realizing image reconstruction and generation at any resolution. (4) The INR encoder outputs the color feature value R corresponding to the spatial coordinates.

7. The image generation method with hybrid linear layer enhancement according to claim 6, characterized in that: The sub-step (2) for obtaining color feature values ​​also includes a super-attention processing step for text information. The processing method of the super-attention processing step is as follows: The original weight matrix After low-rank decomposition or X-factor decomposition, the outer product of two or more low-dimensional vectors is obtained. A word-based super-attention module calculates the importance weight of each word in the text information, selects the most relevant word vectors for fusion, and obtains the semantic importance of each word in the input text. Then, the corresponding low-rank hybrid weights are dynamically selected and fused to generate the attention hybrid weights. This causes the model to focus on the words most relevant to the current image region, using a mixed attention weighting. The calculation formula is: in, It is the initial weight matrix of the current layer. It is a set of word-level mixed tensors. for The transpose of the matrix, It normalizes word similarity using a sentence-level hybrid tensor, generating attention weights. It is a weighted fusion of word-level hybrid tensors. The number of input features, The number of output features, The final weight matrix of the hybrid processing It consists of two parts, the first part being the weights. , derived from sentence-level hybrid tensors, through noise vectors z and text embedding matrix Generation; the second part is the attention-mixed weights. , a word-level hybrid tensor generated from the WHAtt mechanism.

8. A hybrid linear layer enhancement INR-GAN image generation system, used to implement the hybrid linear layer enhancement image generation method according to any one of claims 1-7, characterized in that: The hybrid linear layer enhanced INR-GAN image generation system includes: Visual information processing module: used to perform tagging processing on the acquired visual information, extract the original features of the visual information, superimpose noise on the original features, and output noise-enhanced residual visual tags; Text information processing module: used to encode the input text information and obtain the weight parameters of the INR encoder through the hypernetwork. INR is a hidden neural representation. Color feature acquisition module: Used to map the coordinate information of visual information using the hybrid linear layer and activation function of the INR encoder to obtain the color feature value of the corresponding spatial coordinates; Image synthesis module: It is used to fuse the color features generated in the color feature acquisition step with the noise residual labels obtained in the visual information processing step and input them into the multilayer perceptron network to synthesize an image with high continuity.

9. The INR-GAN image generation system with hybrid linear layer enhancement according to claim 8, characterized in that: The visual information processing module is implemented using deep learning convolutional neural networks (CNNs). The text information processing module includes a text information encoding submodule and a weight parameter acquisition submodule. The text information encoding submodule uses a hypernetwork structure to acquire weight parameters, which are used to construct an implicit neural representation (INR) model to perform subsequent image generation tasks. Based on these weight parameters, text information is dynamically mapped into the structure of the image generation network. The color feature acquisition step is implemented using an INR encoder. A hybrid linear layer is introduced into the backbone network of the INR encoder to inject text conditional information into the image generation process in a parameter mixing manner.

10. The INR-GAN image generation system with hybrid linear layer enhancement according to claim 9, characterized in that: The color feature acquisition module employs the implicit neural representation (INR) modeling method to express the image as a continuous spatial function, enabling the model to generate images at arbitrary resolutions. The color feature acquisition module includes the following sub-steps: (1) Initial feature transformation unit of image: used to perform initial feature transformation on the coordinate information of image through the linear mapping layer of hybrid linear layer to obtain the preliminary features of the transformed coordinate information; (2) Feature fusion unit: used to blend the preliminary features of coordinate information with text information to obtain the blended attention features; (3) Color feature value acquisition unit: used to sum all attention features by weight and obtain the color feature value of the final implicit neural representation through the final linear mapping layer; (4) Output unit: used to output the color feature value corresponding to the spatial coordinates.

Images