Image reconstruction methods, apparatus, equipment, and storage media

By extracting detailed information and textual description information from low-resolution images, and combining self-attention mechanism and global-local attention fusion strategy, the imbalance between local details and global semantics in image super-resolution reconstruction is solved, achieving efficient and accurate image reconstruction.

CN122760342BActive Publication Date: 2026-11-17XIAN ORDNANCE IND TECH IND DEV CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611201747.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-08-10
Publication Date
2026-11-17
Estimated Expiration
2046-08-10

AI Technical Summary

Technical Problem

Existing image super-resolution reconstruction techniques struggle to balance local details with global semantics, suffer from high computational complexity, lack text-visual semantic collaborative guidance, and have weak multi-scale feature fusion adaptive capabilities, leading to problems such as image blurring, texture distortion, and inconsistencies in generated content.

Method used

By extracting detailed information from low-resolution images, multi-scale features are captured using variational autoencoders and parallel convolutional layers. Semantic features are extracted by combining joint self-attention mechanisms and textual description information. A global-local attention weight fusion strategy is adopted to achieve image reconstruction.

Benefits of technology

It improves the visual quality, content accuracy, and algorithm robustness of image reconstruction results, ensures the semantic consistency and detail fidelity of generated images, and reduces computational complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122760342B_ABST
    Figure CN122760342B_ABST
Patent Text Reader

Abstract

This application relates to the field of image processing, and more particularly to an image reconstruction method, apparatus, device, and storage medium. The method extracts detailed information from a low-resolution image to obtain the corresponding latent image representation and shallow image features. Semantic features are extracted based on the latent image representation, shallow image features, and textual description information corresponding to the low-resolution image to obtain corresponding deep image features. The shallow and deep image features are fused to obtain target image features. Image reconstruction is performed based on the target image features to obtain a high-resolution image consistent with the content in the low-resolution image. This application accurately captures the edge and texture details of the image through shallow image features and ensures deep text-visual semantic synergy through deep image features, improving semantic consistency. Image reconstruction based on the fusion results of the two features ensures significant improvements in visual effect, content accuracy, and algorithm robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of image processing, and includes, but is not limited to, an image reconstruction method, apparatus, device, and storage medium. Background Technology

[0002] Image super-resolution reconstruction (SR) is a key task in computer vision, aiming to recover the details and semantic structure of high-resolution (HR) images from low-resolution (LR) images. It is widely used in surveillance, medical imaging, remote sensing, and other scenarios. Current SR technology still faces core challenges: traditional interpolation and early CNN methods struggle to balance local details and global semantics, easily leading to image blurring and texture distortion; while Transformer-based methods improve semantic coherence, they suffer from high computational complexity and insufficient modeling of local details; and while diffusion models enhance the realism of reconstructions, they lack effective text-visual semantic coordination guidance, and multi-scale feature fusion often uses fixed weights, resulting in weak adaptability. Summary of the Invention

[0003] In view of this, the image reconstruction method, apparatus, device, and storage medium provided in the embodiments of this application improve the visual effect, content accuracy, and algorithm robustness of the image reconstruction results.

[0004] The image reconstruction method, apparatus, device, and storage medium provided in this application are implemented as follows: One aspect of this application provides an image reconstruction method, the method comprising: Extract detailed information from low-resolution images to obtain the corresponding latent image representation and shallow image features; Semantic features are extracted based on the latent representation of the image, shallow image features, and text description information corresponding to the low-resolution image to obtain the corresponding deep image features; Feature fusion is performed on shallow image features and deep image features to obtain target image features; Image reconstruction is performed based on the features of the target image to obtain a high-resolution image that is consistent with the content in the low-resolution image.

[0005] In one possible implementation, detailed information from the low-resolution image is extracted to obtain the corresponding latent image representation and shallow image features, including: Image encoding of a resolution image is performed using a variational autoencoder to obtain the corresponding latent image representation; The first convolutional feature and the second convolutional feature are obtained by performing convolution processing on the latent representation of the image through two parallel convolutional layers respectively. By performing feature interaction and feature fusion on the first and second convolutional features, shallow image features are obtained.

[0006] In one possible implementation, feature interaction and feature fusion are performed on the first and second convolutional features to obtain shallow image features, including: Feature activation is performed on the first convolutional feature and the second convolutional feature respectively, and the activation results are cross-multiplied element-wise to obtain the first enhanced feature and the second enhanced feature; The first and second enhanced features are merged and fused, and attention enhancement is performed through the strip multi-head perception module to obtain the first fused feature; The first fused feature and the latent image representation are residually connected to obtain shallow image features.

[0007] In one possible implementation, semantic features are extracted based on the image latent representation, shallow image features, and textual description information corresponding to the low-resolution image to obtain the corresponding deep image features, including: The representation stream, noise stream, and text stream are determined based on the latent image representation and text description information, respectively. The second fusion feature is obtained by performing a preset number of iterations based on the representation stream, noise stream, and text stream using a joint self-attention mechanism. Depth image features are determined based on the second fusion feature and the shallow image features after feature enhancement.

[0008] In one possible implementation, the representation stream, noise stream, and text stream are determined based on the image latent representation and text description information, including: The latent representation of the image is flattened into a sequence of patches, and the representation stream is obtained through linear layer projection and sinusoidal position embedding. Initialize the noise sequence and flatten it into a patch sequence, then add sinusoidal position embedding and time step embedding to obtain the noise stream; The text encoder obtained through pre-training encodes the text description information, and the text stream is obtained through linear layer projection and sinusoidal position embedding.

[0009] In one possible implementation, shallow image features and deep image features are fused to obtain target image features, including: The first concatenated feature is obtained by concatenating the shallow image features and the deep image features; The first concatenated feature is processed with global attention, and the result is simplified to obtain the second concatenated feature. By using a group of convolution kernels with two different convolution scales, two local attention weight maps are obtained for shallow image features and deep image features, respectively. The local attention weights are determined based on the two local attention weight maps, and the second stitched features are weighted based on the local attention weights to obtain the target image features.

[0010] In one possible implementation, image reconstruction is performed based on the features of the target image to obtain a high-resolution image that is consistent with the content in the low-resolution image, including: The target image features are preprocessed, upsampled, enhanced, and normalized to obtain a high-resolution image that is consistent with the content in the low-resolution image.

[0011] Another aspect of the embodiments of this application also provides an image reconstruction apparatus, the apparatus comprising: The edge and detail perception module is used to extract detail information from low-resolution images to obtain the corresponding latent image representation and shallow image features; The semantic extraction module is used to extract semantic features based on the latent representation of the image, shallow image features, and text description information corresponding to the low-resolution image, so as to obtain the corresponding deep image features. The dynamic fusion module is used to fuse shallow and deep image features to obtain target image features; The image reconstruction module is used to reconstruct the image based on the features of the target image, so as to obtain a high-resolution image that is consistent with the content in the low-resolution image.

[0012] In one possible implementation, the edge and detail awareness module is further used for: Image encoding of a resolution image is performed using a variational autoencoder to obtain the corresponding latent image representation; The first convolutional feature and the second convolutional feature are obtained by performing convolution processing on the latent representation of the image through two parallel convolutional layers respectively. By performing feature interaction and feature fusion on the first and second convolutional features, shallow image features are obtained.

[0013] In one possible implementation, the edge and detail awareness module is further used for: Feature activation is performed on the first convolutional feature and the second convolutional feature respectively, and the activation results are cross-multiplied element-wise to obtain the first enhanced feature and the second enhanced feature; The first and second enhanced features are merged and fused, and attention enhancement is performed through the strip multi-head perception module to obtain the first fused feature; The first fused feature and the latent image representation are residually connected to obtain shallow image features.

[0014] In one possible implementation, the semantic extraction module is further used for: The representation stream, noise stream, and text stream are determined based on the latent image representation and text description information, respectively. The second fusion feature is obtained by performing a preset number of iterations based on the representation stream, noise stream, and text stream using a joint self-attention mechanism. Depth image features are determined based on the second fusion feature and the shallow image features after feature enhancement.

[0015] In one possible implementation, the semantic extraction module is further used for: The latent representation of the image is flattened into a sequence of patches, and the representation stream is obtained through linear layer projection and sinusoidal position embedding. Initialize the noise sequence and flatten it into a patch sequence, then add sinusoidal position embedding and time step embedding to obtain the noise stream; The text encoder obtained through pre-training encodes the text description information, and the text stream is obtained through linear layer projection and sinusoidal position embedding.

[0016] In one possible implementation, the dynamic fusion module is further used for: The first concatenated feature is obtained by concatenating the shallow image features and the deep image features; The first concatenated feature is processed with global attention, and the result is simplified to obtain the second concatenated feature. By using a group of convolution kernels with two different convolution scales, two local attention weight maps are obtained for shallow image features and deep image features, respectively. The local attention weights are determined based on the two local attention weight maps, and the second stitched features are weighted based on the local attention weights to obtain the target image features.

[0017] In one possible implementation, the image reconstruction module is further used for: The target image features are preprocessed, upsampled, enhanced, and normalized to obtain a high-resolution image that is consistent with the content in the low-resolution image.

[0018] The electronic device provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the method described in this application.

[0019] The computer-readable storage medium provided in this application embodiment stores a computer program thereon, which, when executed by a processor, implements the method provided in this application embodiment.

[0020] In some embodiments, the method extracts detailed information from a low-resolution image to obtain the corresponding latent image representation and shallow image features. Semantic features are extracted based on the latent image representation, shallow image features, and the corresponding text description information of the low-resolution image to obtain the corresponding deep image features. The shallow and deep image features are fused to obtain the target image features. Image reconstruction is performed based on the target image features to obtain a high-resolution image consistent with the content in the low-resolution image. This application accurately captures the edge and texture details of the image through shallow image features and ensures deep text-visual semantic collaboration through deep image features, improving semantic consistency. Furthermore, image reconstruction is performed based on the fusion results of the two features, ensuring significant improvements in visual effect, content accuracy, and algorithm robustness. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating an image reconstruction method according to an embodiment of this application is shown; Figure 2 This diagram illustrates the structure of an edge and detail sensing module according to an embodiment of this application. Figure 3 This diagram illustrates the structure of a dynamic fusion module according to an embodiment of this application. Figure 4 A schematic diagram of an image reconstruction system according to an embodiment of this application is shown; Figure 5 A schematic diagram of an image reconstruction apparatus according to an embodiment of this application is shown; Figure 6 A schematic diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0025] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0026] It should be noted that the terms "first, second, third" used in the embodiments of this application are used to distinguish similar or different objects and do not represent a specific order of objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0027] The image reconstruction method of this application embodiment can be executed by any electronic device, including but not limited to mobile phones, wearable devices (such as smartwatches, smart bracelets, smart glasses, etc.), tablet computers, laptops, in-vehicle terminals, PCs (Personal Computers), etc. The function implemented by this method can be implemented by the processor in the electronic device calling program code. Of course, the program code can be stored in a computer storage medium. It can be seen that the electronic device includes at least a processor and a storage medium.

[0028] The image reconstruction method of this application can be used in any application scenario that requires reconstructing a high-resolution image from a low-resolution image. For example, this application can be applied to the application scenario of repairing and reconstructing blurry face images in the field of security and public safety. Alternatively, it can also be applied to the application scenario of repairing and reconstructing low-resolution remote sensing images in the field of remote sensing and geographic mapping.

[0029] Current related technologies have the following main technical shortcomings when performing image reconstruction: The imbalance between local details and global semantics is a defect that manifests itself in the fact that although traditional interpolation algorithms and early CNN-based methods are fast, their receptive fields are limited, making it difficult to capture long-distance dependent global semantic information. This results in blurry reconstruction results, texture distortion, or artifacts.

[0030] The Transformer method suffers from computational bottlenecks and a lack of detail. Specifically, while Transformer-based methods improve the semantic coherence of features through self-attention mechanisms, their high computational complexity leads to a quadratic increase in the computational cost of the attention mechanism when processing high-resolution images, making them difficult to deploy in practical applications. Furthermore, insufficient local modeling causes the Transformer's global attention mechanism to focus on the overall picture, resulting in relatively weak modeling capabilities for pixel-level local details (such as fine lines and complex texture patterns).

[0031] The shallow nature of cross-modal fusion is a specific shortcoming of existing text-guided super-resolution (Cross-modal SR) methods, which mostly remain at the level of shallow feature concatenation. That is, they simply combine text feature vectors and image feature vectors without achieving deep semantic alignment and bidirectional interaction. This makes it difficult for text descriptions to effectively constrain the image generation process, easily leading to "semantic distortion," where the generated image content does not match the text description.

[0032] Feature fusion lacks adaptability. This deficiency manifests itself in the fact that existing methods often employ fixed weights or a single attention mechanism when fusing features at multiple scales (combining features from different levels). Since image content is heterogeneous, flat regions and textured regions have different requirements for feature fusion. Fixed-weight fusion methods cannot dynamically adjust the ratio of global to local information based on image content, resulting in limited expressive power of the fused features.

[0033] The lack of semantic guidance and cost issues in generative models are significant drawbacks. Specifically, while next-generation generative methods, such as diffusion models, can reconstruct realistic textures, they often lack effective text-visual semantic coordination. This leads to a situation where the model is "drawing" but doesn't know "what it's drawing," resulting in realistic content that may not meet user expectations. Furthermore, the high training and inference costs of large-scale models limit their practical application in real-world scenarios.

[0034] Therefore, the technical problem solved by the embodiments of this application is how to improve the visual effect, content accuracy and algorithm robustness of image reconstruction results.

[0035] The image reconstruction scheme of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0036] Figure 1 A flowchart illustrating an image reconstruction method according to an embodiment of this application is shown. Figure 1 As shown, the image reconstruction method of this application embodiment may include the following steps S10-S40.

[0037] For ease of description, the image reconstruction method of this application embodiment is described using an electronic device as the execution subject. It should be understood that the execution subject of this application embodiment can also be a processor or chip in an electronic device, and this application embodiment does not impose any limitations.

[0038] Step S10: Extract detailed information from the low-resolution image to obtain the corresponding latent image representation and shallow image features.

[0039] In one possible implementation, the electronic device receives a low-resolution image that needs to be reconstructed. It can also receive text description information corresponding to the low-resolution image, which can be uploaded by a user or sent to the electronic device via other devices. The content included in the low-resolution image varies depending on the application scenario; for example, it can be a face image, an environmental image, or a remote sensing image.

[0040] Optionally, the text description information corresponding to the low-resolution image can be obtained through user upload or by directly recognizing the image content. For example, the text description information can be the image processing requirements input by the user, such as "Reconstruct this blurry face into a clear version, requiring fine skin texture and bright eyes." Alternatively, it can also include a description of the image content obtained by extracting image information, such as "An orange and white cat with glossy, smooth fur, sitting on green grass."

[0041] In some embodiments, after acquiring a low-resolution image, the electronic device extracts detailed information such as edge features and texture features from the low-resolution image to obtain the corresponding latent image representation and shallow image features. Optionally, the extraction process of the shallow image features may include image encoding of the low-resolution image using a variational autoencoder to obtain the corresponding latent image representation. The latent image representation is then convolved using two parallel convolutional layers to obtain a first convolutional feature and a second convolutional feature. Feature interaction and feature fusion are performed on the first and second convolutional features to obtain the shallow image features. The process of the electronic device performing feature interaction and feature fusion on the first and second convolutional features to obtain the shallow image features may further include: performing feature activation on the first and second convolutional features respectively, and performing element-wise cross-multiplication on the activation results to obtain a first enhanced feature and a second enhanced feature. The first enhanced feature and the second enhanced feature are merged and fused, and attention enhancement is performed through a striped multi-head perception module to obtain a first fused feature. The first fused feature and the latent image representation are residually concatenated to obtain the shallow image features.

[0042] Figure 2 A schematic diagram of the structure of an edge and detail sensing module according to an embodiment of this application is shown. Figure 2As shown, after acquiring a low-resolution image I, the electronic device first encodes the low-resolution image using a pre-trained variational autoencoder, preserving its main structural information to obtain the latent image representation I. LR .

[0043] After obtaining the latent image representation, it is convolved by two parallel convolutional layers with different scales. This convolution process is used to capture local features and longer-range dependencies. The first convolutional layer uses a 3×3 kernel to capture fine-grained local patterns in the data, which can determine the first convolutional feature as... The second convolutional layer uses a larger 7×7 kernel to extract features and identify broader, longer-range dependencies. This can be achieved by determining the features of the second convolutional layer as... .

[0044] Furthermore, the electronic device applies GELU activation to the first and second convolutional features output from the two convolutions respectively, and performs element-wise cross-multiplication to enhance feature interaction, thus obtaining the first enhanced feature. Second Enhancement Features .

[0045] Finally, the electronic device can merge and add the two interacting first and second enhanced features, and then fuse them through a 1×1 convolution to further extract new features. These features are then residually connected to the input features through a striped multi-head perception module (Att) to output shallow image features. .

[0046] The strip multi-head perception module performs channel and spatial compression through two convolutional layers. The first convolutional layer uses a 1×1 kernel to compress the number of channels in the input feature map to half, reducing computational complexity while preserving important feature information. The second convolutional layer employs grouped convolution with a larger kernel, further expanding the number of channels to twice the original number, resulting in feature X. sp After the compression process described above, the feature map is first divided into two parts: a top branch and a bottom branch. The top branch divides the image into several horizontal strips, and the bottom branch divides the image into several vertical strips, each strip having a height of 2 pixels. A multi-head attention mechanism is then applied to each horizontal and vertical strip to obtain the feature map S. h and S v .

[0047] For multi-head attention mechanisms, taking horizontal stripes as an example, firstly, the batch size and shape information of the stripes are obtained. To prepare for subsequent attention calculations, the last dimension of the stripe is moved to the second dimension. Next, three linear layers are used to perform linear transformations on the query (Q), key (K), and value (V), respectively. , Let Q, K, and V be the values ​​of the i-th attention head. This is the corresponding weight matrix. These linear layers transform the number of channels in the input data into the number of heads multiplied by the hidden size corresponding to each head. Then, the transformed data is reshaped. To compute the dot product attention, the last and second-to-last dimensions of K are transposed. Next, the dot product of Q and the transposed K is calculated to obtain the attention score matrix. Since the scores of the dot product attention can be very large, this may cause the softmax function to enter the saturation region, resulting in a gradient close to zero, so the dot product scores need to be scaled. Here, scaling is achieved by dividing by the square root of the feature dimension. Then, the softmax function is applied to the scaled scores to obtain the normalized attention weights. The attention weights are multiplied by the value V to obtain a weighted output, achieved through matrix multiplication, where each row of the attention weight matrix corresponds to a weighting coefficient in V. Finally, the data is reshaped using another linear layer to restore its shape to match the shape of the original input, resulting in Q. v The same process applies to vertical strip branching, but it is performed in the width direction. h After channel attention and S v Element-wise multiplication is performed and a new feature map Q is generated through 1×1 convolution. h The formula is , .

[0048] Therefore, it can be seen that in the process of shallow image feature extraction, the embodiments of this application achieve multi-scale coverage through parallel asymmetric convolution, feature purification through cross-multiplication, precise enhancement of directional details through strip attention, and finally, information integrity through residual connections. This series of operations ensures that the output shallow image features are high-quality feature maps with rich details, clear structure, and suppressed noise.

[0049] Step S20: Extract semantic features based on the latent representation of the image, the shallow image features, and the text description information corresponding to the low-resolution image to obtain the corresponding deep image features.

[0050] In one possible implementation, after the electronic device completes shallow feature extraction of the low-resolution image, it performs semantic feature extraction together with the image latent representation and shallow image features obtained from the shallow image feature extraction process and the corresponding text description information of the low-resolution image to obtain the corresponding deep image features. Specifically, the electronic device can first determine the representation stream, noise stream, and text stream based on the image latent representation and text description information, respectively. Then, it performs a preset number of iterations based on the representation stream, noise stream, and text stream using a joint self-attention mechanism to obtain a second fused feature. Finally, based on the second fused feature and the shallow image features after feature enhancement processing, the deep image features are determined.

[0051] Optionally, the process of determining the representation stream, noise stream, and text stream is used to transform the image structure, random noise, and text semantics into a sequence format that the Transformer can understand. Specifically, the electronic device can flatten the latent image representation into a sequence of patches, and obtain the representation stream through linear layer projection and sinusoidal position embedding. That is, I... LR Flattened into a sequence of patches to fit the converter's input format, the patch size can be... The number of patches can be Each patch is projected into a D-dimensional embedding space through a linear layer. To preserve spatial information, a sinusoidal position embedding is further added to the projection result to obtain the representation stream. ,in, The spatial location of the encoded patch is generated using sine and cosine functions. The representation stream provides structural information about the low-resolution image, guiding the diffusion process to generate high-resolution features consistent with the input image.

[0052] Optionally, the electronic device can initialize the noise sequence and flatten it into a patch sequence, adding sinusoidal position embeddings and time step embeddings to obtain a noise stream. That is, the noise stream can be initialized as Gaussian noise. , where t is the time step of the diffusion process, representing the intermediate representation of gradual denoising from the initial noise. Let Z t Flatten into a patch sequence Where the X dimension is ,in D is the embedding dimension. Add sinusoidal position embedding and time step embedding to X. , The current time step t is encoded using sinusoidal coding or learnable embeddings. The noise stream is the core of the diffusion model generation process, gradually transforming random noise into image features, with time-step embeddings guiding the denoising progress.

[0053] Optionally, the electronic device can use a pre-trained text encoder to encode the text description information, and obtain a text stream through linear layer projection and sinusoidal position embedding. The text stream is then processed by three pre-trained text encoders, CLIP-L and T5XXL, to generate semantically rich text tags. CLIP-L is used to generate pooled semantic embeddings based on the text description caption. T5XXL generates detailed text tag sequences. Then, the encoder output is projected onto a unified embedding dimension through a linear layer. Adding sinusoidal position embeddings to C encodes the text marker sequence positions to obtain a text stream. In this embodiment, the text stream provides semantic guidance, enhancing the semantic consistency and realism of the generated images.

[0054] In some embodiments, after determining the representation stream, noise stream, and text stream, the electronic device performs a predetermined number of iterations based on these streams using a joint self-attention mechanism to obtain a second fused feature. Then, based on the second fused feature and the shallow image features after feature enhancement, a depth image feature is determined. During each iteration, the electronic device can input the noise stream, representation stream, and text stream into a joint self-attention mechanism to achieve bidirectional interaction between the three streams. This interaction process first calculates the query... ,key Sum ,in, It is a linear projection matrix. This represents the concatenation operation. Next, attention weights are calculated to obtain the fused features. , where d is the scaling factor. Furthermore, the electronic device continues to input the fused features into the feedforward network (FFN) and layer normalization (LN) for stabilization training. The formula for this stabilization training process can be... and During training, the electronic device does not completely retrain all parameters; instead, it uses a low-rank update matrix. This means that only a small increment, calculated from the low-rank matrix, is added to the original pre-trained weights. This significantly reduces the number of parameters that need to be updated and the computational cost.

[0055] Optionally, in this embodiment, a semantic extraction module consisting of a preset number of joint self-attention mechanisms, feedforward networks (FFNs) and layer normalization (LNs) is stacked. After the electronic device performs a preset number of iterations on the representation stream, noise stream, and text stream, it can convert the original noise stream sequence into a high-quality feature sequence that integrates text semantics and image structure. This sequence is then reshaped back into the format of a feature map to obtain the second fused feature.

[0056] Furthermore, after determining the second fusion feature, the electronic device further processes the shallow image feature X1 through a 3×3 convolutional layer to obtain... Where AB is a low-rank matrix. Further, the electronic device determines the deep image features based on the processed shallow image features and the second fused features. ,in It is a low-rank update matrix. These are low-rank parameters. While the second fused feature obtained after multiple Transformer layers is semantically rich, its precise pixel location information may be somewhat blurred. The 3×3 convolution leverages its local perception capability to re-inject precise pixel-level details into the features.

[0057] Therefore, it can be seen that in the process of deep image feature extraction, the embodiments of this application achieve depth alignment between text and image through three-stream joint attention, achieve semantically controllable detail generation through a diffusion mechanism, achieve efficient optimization through low-rank updates, and finally ensure the unity of semantics and accuracy through pixel-level enhancement. This series of designs results in a high-quality feature map of the output deep image that is structurally accurate, semantically rich, detail-sharp, and computationally efficient.

[0058] Step S30: Perform feature fusion on the shallow image features and the deep image features to obtain the target image features.

[0059] In one possible implementation, after determining shallow and deep image features, the electronic device fuses these features to obtain target image features. This process combines features from two different dimensions to adaptively adjust the fusion weights. Specifically, the feature fusion process may include concatenating the shallow and deep image features to obtain a first concatenated feature. Global attention processing is then applied to the first concatenated feature, and the result is simplified to obtain a second concatenated feature. Two local attention weight maps are obtained for the shallow and deep image features respectively using a set of convolutional kernels with two different convolutional scales. Local attention weights are determined based on the two local attention weight maps, and the second concatenated feature is weighted based on these local attention weights to obtain the target image features.

[0060] Figure 3 A schematic diagram of the structure of a dynamic fusion module according to an embodiment of this application is shown. Figure 3 As shown, during the feature fusion process, the electronic device can first combine the shallow image features X1 from the edge and detail perception module and the deep image features from the semantic feature extraction module. (X2) Concatenate the two features in the first channel dimension to form a new first concatenated feature. This step combines the two features through a simple concatenation operation, laying the foundation for subsequent fusion operations. The concatenation formula can be... Furthermore, the electronic device performs adaptive average pooling on the concatenated second feature output to obtain a global feature of size 1×1. Adaptive average pooling can extract global information from the feature map, which helps in the subsequent calculation of global attention weights.

[0061] Furthermore, a global attention weight `att` is calculated using a 2D convolutional layer and a sigmoid activation function. Specifically, the global features are processed through a 2D convolutional layer to extract important information, and then normalized to the range of 0 to 1 using a sigmoid activation function to obtain the global attention weight. This global attention weight is then applied to the concatenated feature map for weighted calculation. This can enhance important parts of the feature map. Then, a 2D convolutional layer and activation function are used to reduce the number of channels in the feature map from 2 dimensions to 1 dimension to obtain the second concatenated feature. This step simplifies the complexity of the feature map through dimensionality reduction, which is helpful for subsequent calculations and processing.

[0062] After determining the second concatenation feature, the shallow image feature X1 and the deep image feature X2 are processed by convolutional kernel groups consisting of two 1×1 and 3×3 convolutional layers, respectively, to obtain two local attention weight maps. and 2D convolutional layers can capture detailed information about feature maps in both spatial and temporal dimensions, thereby calculating more accurate attention weight maps.

[0063] Finally, the electronic device can add the two local attention weight maps together and then calculate the final local attention weight `att` using the Sigmoid activation function. Local attention weights can more precisely focus on the details in the feature maps, making the feature fusion process more refined. Finally, the local attention weights are applied to the output feature map `output`, weighted, and the weighted output feature map is returned as the target image feature. Through the combined action of global and local attention mechanisms, input features and skip connection features are effectively fused, and the output feature map not only contains global information but also retains local details, thereby improving feature representation capabilities.

[0064] Therefore, it can be seen that in the feature fusion process, this application embodiment controls the overall structure through global attention, focuses on subtle differences through local attention, and improves efficiency through channel dimensionality reduction, ultimately achieving adaptive and dynamic balance fusion of shallow details and deep semantics. This design ensures that the output target image features are high-quality feature representations that understand both the overall picture and the local details, maintaining both accuracy and efficiency, laying a solid foundation for the final success of the image reconstruction module.

[0065] Step S40: Reconstruct the image based on the features of the target image to obtain a high-resolution image that is consistent with the content in the low-resolution image.

[0066] In one possible implementation, after determining the features of the target image, the electronic device can perform image reconstruction based on the target image features to obtain a high-resolution image that is consistent with the content in the low-resolution image. This image reconstruction process can involve feature preprocessing, upsampling, feature enhancement, and normalization of the target image features to obtain a high-resolution image consistent with the content in the low-resolution image.

[0067] Optionally, the electronic device can input the fused target image features into a 1×1 convolutional layer and apply a linear activation function. This step aims to preprocess the features, preparing them for subsequent upsampling. The target image features are then magnified using convolutional layers and pixel rearrangement techniques. Pixel rearrangement effectively maps features to a higher spatial resolution, laying the foundation for final image reconstruction. A 1×1 convolution operation is applied again to the upsampled target image features to further extract and enhance the feature representation, preparing for the generation of a high-quality, high-resolution image. The processed target image features undergo an appropriate normalization step, typically limiting pixel values ​​to a suitable range, such as [0,1] or [-1,1]. Furthermore, a mean or other correction term is added to generate the final high-resolution image. This step ensures that the output image visually approximates a true high-resolution image while preserving the fused multi-scale features.

[0068] Figure 4 A schematic diagram of an image reconstruction system according to an embodiment of this application is shown. Figure 4 As shown, the steps of the image reconstruction method in this embodiment can be executed by integrated algorithm modules. These modules can be integrated into an image reconstruction system, which can then be directly invoked by the electronic device to perform image reconstruction. The image reconstruction system may include an edge and detail perception module, a semantic extraction module, a dynamic fusion module, and an image reconstruction module.

[0069] Optionally, after acquiring a low-resolution image and its corresponding text description, the electronic device can extract detailed information from the low-resolution image using an edge and detail perception module to obtain the corresponding latent image representation and shallow image features. A semantic extraction module then extracts semantic features based on the latent image representation, shallow image features, and the corresponding text description information of the low-resolution image to obtain the corresponding deep image features. A dynamic fusion module then fuses the shallow and deep image features to obtain the target image features. Finally, an image reconstruction module performs image reconstruction based on the target image features to obtain a high-resolution image consistent with the content of the low-resolution image.

[0070] Based on the aforementioned technical features, the adaptive super-resolution method based on text-visual semantic collaborative enhancement proposed in this application achieves deep alignment and bidirectional interaction between text semantics and image structure by constructing a three-stream joint self-attention mechanism, significantly improving the semantic consistency and content fidelity of the reconstructed image. Simultaneously, the edge and detail perception module employs parallel asymmetric convolution and striped multi-head attention to accurately capture multi-scale directional details, effectively solving the problems of edge blurring and texture distortion. Furthermore, the dynamic fusion module, through a global-local dual-attention adaptive weighting strategy, achieves a dynamic balance between shallow detail features and deep semantic features, overcoming the limitations of fixed-weight fusion. Overall, this method systematically solves the core problems in existing super-resolution technologies, such as the imbalance between detail fidelity and semantic consistency, and insufficient flexibility in feature fusion, ultimately achieving high-resolution image reconstruction with both realistic visual quality and rich semantic structure.

[0071] It should be understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0072] Based on the foregoing embodiments, this application provides an image reconstruction device, which includes various modules and units included in each module, and can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0073] Figure 5 A schematic diagram of an image reconstruction apparatus according to an embodiment of this application is shown. Figure 5 As shown, the image reconstruction apparatus of this application embodiment includes: The edge and detail perception module 50 is used to extract detail information from low-resolution images to obtain the corresponding latent image representation and shallow image features. The semantic extraction module 51 is used to extract semantic features based on the latent representation of the image, shallow image features, and text description information corresponding to the low-resolution image to obtain the corresponding deep image features. The dynamic fusion module 52 is used to fuse shallow image features and deep image features to obtain target image features; Image reconstruction module 53 is used to reconstruct the image based on the features of the target image to obtain a high-resolution image that is consistent with the content in the low-resolution image.

[0074] In one possible implementation, the edge and detail awareness module 50 is further used for: Image encoding of a resolution image is performed using a variational autoencoder to obtain the corresponding latent image representation; The first convolutional feature and the second convolutional feature are obtained by performing convolution processing on the latent representation of the image through two parallel convolutional layers respectively. By performing feature interaction and feature fusion on the first and second convolutional features, shallow image features are obtained.

[0075] In one possible implementation, the edge and detail awareness module 50 is further used for: Feature activation is performed on the first convolutional feature and the second convolutional feature respectively, and the activation results are cross-multiplied element-wise to obtain the first enhanced feature and the second enhanced feature; The first and second enhanced features are merged and fused, and attention enhancement is performed through the strip multi-head perception module to obtain the first fused feature; The first fused feature and the latent image representation are residually connected to obtain shallow image features.

[0076] In one possible implementation, the semantic extraction module 51 is further used for: The representation stream, noise stream, and text stream are determined based on the latent image representation and text description information, respectively. The second fusion feature is obtained by performing a preset number of iterations based on the representation stream, noise stream, and text stream using a joint self-attention mechanism. Depth image features are determined based on the second fusion feature and the shallow image features after feature enhancement.

[0077] In one possible implementation, the semantic extraction module 51 is further used for: The latent representation of the image is flattened into a sequence of patches, and the representation stream is obtained through linear layer projection and sinusoidal position embedding. Initialize the noise sequence and flatten it into a patch sequence, then add sinusoidal position embedding and time step embedding to obtain the noise stream; The text encoder obtained through pre-training encodes the text description information, and the text stream is obtained through linear layer projection and sinusoidal position embedding.

[0078] In one possible implementation, the dynamic fusion module 52 is further used for: The first concatenated feature is obtained by concatenating the shallow image features and the deep image features; The first concatenated feature is processed with global attention, and the result is simplified to obtain the second concatenated feature. By using a group of convolution kernels with two different convolution scales, two local attention weight maps are obtained for shallow image features and deep image features, respectively. The local attention weights are determined based on the two local attention weight maps, and the second stitched features are weighted based on the local attention weights to obtain the target image features.

[0079] In one possible implementation, the image reconstruction module 53 is further used for: The target image features are preprocessed, upsampled, enhanced, and normalized to obtain a high-resolution image that is consistent with the content in the low-resolution image.

[0080] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0081] It should be noted that, in the embodiments of this application... Figure 5 The module division of the image reconstruction device shown is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or be integrated into one unit by two or more units. The integrated units can be implemented in hardware, as software functional units, or a combination of software and hardware.

[0082] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0083] Figure 6 A schematic diagram of an electronic device according to an embodiment of this application is shown. For example... Figure 6 As shown in the figure, this application provides an electronic device, which can be a server, and its internal structure diagram can be as follows. Figure 6 As shown, the electronic device includes a processor 620, a memory, and a transceiver 640 connected via a system bus 610. The processor 620 provides computing and control capabilities. The memory includes a non-volatile storage medium 631 and internal memory 632. The non-volatile storage medium 631 stores an operating system, computer programs, and a database. The internal memory 632 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium 631. The database stores data. The transceiver 640 communicates with external terminals via a network connection. The computer program, when executed by the processor 620, implements the methods described above.

[0084] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor 620, implements the steps of the method provided in the above embodiments.

[0085] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0086] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0087] In one possible implementation, the image reconstruction apparatus provided in this application can be implemented as a computer program, which can be implemented in, for example... Figure 6 The device operates on the electronic device shown. The memory of the electronic device can store the various program modules that make up the above-described apparatus. The computer program composed of the various program modules causes the processor 620 to execute the steps of the methods in the various embodiments of this application described in this specification.

[0088] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0089] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, phrases such as "in one possible implementation," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0090] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0091] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0093] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0094] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0095] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0096] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0097] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0098] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0099] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0100] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of image reconstruction, characterized by, The method includes: Extract detailed information from low-resolution images to obtain the corresponding latent image representation and shallow image features; Semantic features are extracted based on the latent representation of the image, the shallow image features, and the text description information corresponding to the low-resolution image to obtain the corresponding deep image features; The shallow image features and the deep image features are fused to obtain the target image features; Image reconstruction is performed based on the target image features to obtain a high-resolution image that is consistent with the content in the low-resolution image; The step of extracting detailed information from low-resolution images to obtain corresponding latent image representations and shallow image features includes: The resolution image is encoded using a variational autoencoder to obtain the corresponding latent image representation. The image latent representation is processed by two parallel convolutional layers to obtain the first convolutional feature and the second convolutional feature. The first convolutional feature and the second convolutional feature are subjected to feature interaction and feature fusion to obtain shallow image features; The step of performing feature interaction and feature fusion on the first convolutional features and the second convolutional features to obtain shallow image features includes: Feature activation is performed on the first convolutional feature and the second convolutional feature respectively, and the activation results are cross-multiplied element-wise to obtain the first enhanced feature and the second enhanced feature; The first enhanced feature and the second enhanced feature are merged and fused, and attention enhancement is performed through the strip multi-head perception module to obtain the first fused feature; The first fused feature and the latent image representation are residually connected to obtain shallow image features; The step of extracting semantic features based on the latent image representation, the shallow image features, and the text description information corresponding to the low-resolution image to obtain the corresponding deep image features includes: The representation stream, noise stream, and text stream are determined based on the latent image representation and the text description information, respectively. The second fusion feature is obtained by performing a preset number of iterations based on the representation stream, noise stream, and text stream using a joint self-attention mechanism. Based on the second fusion feature and the shallow image feature after feature enhancement processing, the deep image feature is determined.

2. The method of claim 1, wherein, The step of determining the representation stream, noise stream, and text stream based on the image latent representation and the text description information includes: The latent representation of the image is flattened into a sequence of patches, and a representation stream is obtained through linear layer projection and sinusoidal position embedding. Initialize the noise sequence and flatten it into a patch sequence, then add sinusoidal position embedding and time step embedding to obtain the noise stream; The text encoder obtained through pre-training encodes the text description information, and the text stream is obtained through linear layer projection and sinusoidal position embedding.

3. The method of claim 1, wherein, The feature fusion of the shallow image features and the deep image features to obtain the target image features includes: The shallow image features and the deep image features are concatenated to obtain the first concatenated feature; The first splicing feature is subjected to global attention processing, and the processing result is simplified to obtain the second splicing feature; Two local attention weight maps are obtained by using a group of convolution kernels including two convolution kernels with different convolution scales for the shallow image features and the deep image features, respectively. The local attention weights are determined based on the two local attention weight maps, and the second stitching features are weighted based on the local attention weights to obtain the target image features.

4. The method of claim 1, wherein, The step of reconstructing the image based on the target image features to obtain a high-resolution image that is consistent with the content in the low-resolution image includes: The target image features are preprocessed, upsampled, enhanced, and normalized to obtain a high-resolution image that is consistent with the content in the low-resolution image.

5. An image reconstruction apparatus, characterized by comprising: The device includes: The edge and detail perception module is used to extract detail information from low-resolution images to obtain the corresponding latent image representation and shallow image features. The semantic extraction module is used to extract semantic features based on the latent representation of the image, the shallow image features, and the text description information corresponding to the low-resolution image, so as to obtain the corresponding deep image features. The dynamic fusion module is used to fuse the shallow image features and the deep image features to obtain the target image features; An image reconstruction module is used to reconstruct the image based on the features of the target image to obtain a high-resolution image that is consistent with the content in the low-resolution image. The step of extracting detailed information from low-resolution images to obtain corresponding latent image representations and shallow image features includes: The resolution image is encoded using a variational autoencoder to obtain the corresponding latent image representation. The image latent representation is processed by two parallel convolutional layers to obtain the first convolutional feature and the second convolutional feature. The first convolutional feature and the second convolutional feature are subjected to feature interaction and feature fusion to obtain shallow image features; The step of performing feature interaction and feature fusion on the first convolutional features and the second convolutional features to obtain shallow image features includes: Feature activation is performed on the first convolutional feature and the second convolutional feature respectively, and the activation results are cross-multiplied element-wise to obtain the first enhanced feature and the second enhanced feature; The first enhanced feature and the second enhanced feature are merged and fused, and attention enhancement is performed through the strip multi-head perception module to obtain the first fused feature; The first fused feature and the latent image representation are residually connected to obtain shallow image features; The step of extracting semantic features based on the latent image representation, the shallow image features, and the text description information corresponding to the low-resolution image to obtain the corresponding deep image features includes: The representation stream, noise stream, and text stream are determined based on the latent image representation and the text description information, respectively. The second fusion feature is obtained by performing a preset number of iterations based on the representation stream, noise stream, and text stream using a joint self-attention mechanism. Based on the second fusion feature and the shallow image feature after feature enhancement processing, the deep image feature is determined.

6. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • Text image super-resolution enhancement method, electronic equipment and storage medium

    CN119338680A

  • Astronomical image super-resolution reconstruction method and device, equipment and storage medium

    CN119963420A