Image processing method, training method of image reconstruction model and electronic equipment

Abstract

The invention provides an image processing method, a training method of an image reconstruction model and electronic equipment, and relates to the field of artificial intelligence technology and image processing. The method comprises the following steps: acquiring an initial feature map of an original image; performing attention processing on the features of different frequencies in the initial feature map to obtain an enhanced feature map of the original image and an attention weight matrix of the enhanced feature map; the enhanced feature map is adjusted based on an attention weight matrix corresponding to the enhanced feature map, a spatial feature map of the original image is obtained, and the spatial feature map is used for representing correlation between semantic information and spatial positions between different pixel points; and reconstructing the original image based on the spatial feature map to obtain a target image. According to the invention, the technical problem of poor quality of reconstruction of the original image in the prior art is solved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology and the field of image processing. Specifically, it relates to an image processing method, a training method for an image reconstruction model, and an electronic device. Background Technology

[0002] Currently, the most common method for reconstructing original images is image super-resolution. This method performs super-resolution processing on the original image, which can improve the image resolution, restore lost details and textures, and thus improve the visual quality of the image to some extent. However, this reconstruction method achieves this by discarding some feature information or oversimplifying image details. In practical applications, details can still be lost and structures distorted, resulting in the reconstructed target image still having poor image quality.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an image processing method, an image reconstruction model training method, and an electronic device to at least solve the technical problem of poor quality reconstruction of original images in related technologies.

[0005] According to one aspect of the embodiments of this application, an image processing method is provided, comprising: acquiring an initial feature map of an original image, wherein the initial feature map is used to characterize the visual features of different pixels in the original image; performing attention processing on features of different frequencies in the initial feature map to obtain an enhanced feature map of the original image and an attention weight matrix of the enhanced feature map, wherein the enhanced feature map is used to characterize the dependency relationship between different pixels, and the enhanced feature map is constructed based on the attention weight matrix; adjusting the enhanced feature map based on the attention weight matrix corresponding to the enhanced feature map to obtain a spatial feature map of the original image, wherein the spatial feature map is used to characterize the semantic information and spatial location correlation between different pixels; and reconstructing the original image based on the spatial feature map to obtain a target image.

[0006] According to one aspect of the embodiments of this application, a training method for an image reconstruction model is also provided, comprising: acquiring a training image and a reference image; inputting the training image into an initial reconstruction model, and using the initial reconstruction model to reconstruct the training image to obtain a reconstructed image; constructing a model loss function for the initial reconstruction model based on the reconstructed image and the reference image; and adjusting the initial reconstruction model using the model loss function to obtain an image reconstruction model, wherein the image reconstruction model is used in the above-described method.

[0007] According to another aspect of the embodiments of this application, an image processing method is also provided, comprising: upon detecting an input instruction acting on an operation interface, displaying an initial feature map of an original image on the operation interface, wherein the initial feature map is used to characterize the visual features of different pixels in the original image; and upon detecting a processing instruction acting on the operation interface, displaying a target image on the operation interface, wherein the target image is obtained by reconstructing the original image from a spatial feature map, the spatial feature map being used to characterize the semantic information and spatial location correlation between different pixels, the spatial feature map being obtained by adjusting the enhanced feature map using an attention weight matrix corresponding to the enhanced feature map, the enhanced feature map being used to characterize the dependency relationship between different pixels, the enhanced feature map being constructed based on the attention weight matrix, and the enhanced feature map and the attention weight matrix being obtained by performing attention processing on features of different frequencies in the initial feature map.

[0008] According to another aspect of the embodiments of this application, an image processing apparatus is also provided, comprising: a feature map acquisition module, configured to acquire an initial feature map of an original image, wherein the initial feature map is used to characterize the visual features of different pixels in the original image; an attention processing module, configured to perform attention processing on features of different frequencies in the initial feature map to obtain an enhanced feature map of the original image and an attention weight matrix of the enhanced feature map, wherein the enhanced feature map is used to characterize the dependency relationship between different pixels and is constructed based on the attention weight matrix; a feature map adjustment module, configured to adjust the enhanced feature map based on the attention weight matrix corresponding to the enhanced feature map to obtain a spatial feature map of the original image, wherein the spatial feature map is used to characterize the semantic information and spatial location correlation between different pixels; and an image reconstruction module, configured to reconstruct the original image based on the spatial feature map to obtain a target image.

[0009] According to one aspect of the embodiments of this application, a training apparatus for an image reconstruction model is also provided, comprising: an image acquisition module for acquiring a training image and a reference image; an initial reconstruction module for inputting the training image into an initial reconstruction model and reconstructing the training image using the initial reconstruction model to obtain a reconstructed image; a function construction module for constructing a model loss function of the initial reconstruction model based on the reconstructed image and the reference image; and a model adjustment module for adjusting the initial reconstruction model using the model loss function to obtain an image reconstruction model, wherein the image reconstruction model is used in the above-described method.

[0010] According to another aspect of the embodiments of this application, another image processing apparatus is also provided, comprising: a first display module, configured to display an initial feature map of an original image on the operation interface when an input instruction acting on the operation interface is detected, wherein the initial feature map is used to characterize the visual features of different pixels in the original image; and a second display module, configured to display a target image on the operation interface when a processing instruction acting on the operation interface is detected, wherein the target image is obtained by reconstructing the original image from a spatial feature map, the spatial feature map being used to characterize the semantic information and spatial location correlation between different pixels, the spatial feature map being obtained by adjusting the enhanced feature map by an attention weight matrix corresponding to the enhanced feature map, the enhanced feature map being used to characterize the dependency relationship between different pixels, the enhanced feature map being constructed based on the attention weight matrix, and the enhanced feature map and the attention weight matrix being obtained by performing attention processing on features of different frequencies in the initial feature map.

[0011] According to another aspect of the embodiments of this application, a computing device is also provided, including: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0012] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory storing an executable program; and a processor connected to the memory via a bus for running the program, wherein the program executes the methods in various embodiments of this application when it runs.

[0013] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is running, it controls the device where the computer-readable storage medium is located to perform the methods of various embodiments of this application.

[0014] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the methods of various embodiments of this application.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the methods in various embodiments of this application.

[0016] According to another aspect of the embodiments of this application, a computer program is also provided, which, when executed by a processor, implements the methods of the various embodiments of this application.

[0017] In this embodiment, the method involves obtaining an initial feature map of the original image; performing attention processing on features of different frequencies in the initial feature map to obtain an enhanced feature map of the original image and an attention weight matrix for the enhanced feature map; adjusting the enhanced feature map based on the attention weight matrix to obtain a spatial feature map of the original image; and reconstructing the original image based on the spatial feature map to obtain the target image. By performing attention processing on the initial feature map of the original image and then adjusting the enhanced feature map obtained during the attention processing using the attention weight matrix to obtain the spatial feature map, the dependency between different pixels and the correlation between semantic information and spatial location between different pixels can be effectively enhanced. This improves the accuracy of reconstructing the original image using the spatial feature map and effectively improves the image quality of the reconstructed target image, thereby solving the technical problem of poor image quality in related technologies.

[0018] The above general description and the following detailed description are for illustrative and explanatory purposes only and do not constitute a limitation thereof. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a schematic diagram illustrating an application scenario of an image processing method according to an embodiment of this application;

[0021] Figure 2 This is a flowchart illustrating an image processing method according to an embodiment of this application;

[0022] Figure 3 This is a schematic diagram illustrating a first decoupling module according to an embodiment of this application;

[0023] Figure 4 This is a schematic diagram illustrating a second decoupling module according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram illustrating an image reconstruction process according to an embodiment of this application;

[0025] Figure 6 This is a schematic diagram illustrating a comparison of scanning results according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram illustrating a comparison of reconstruction effects according to an embodiment of this application;

[0027] Figure 8This is a schematic diagram illustrating another reconstruction effect comparison result according to an embodiment of this application;

[0028] Figure 9 This is a flowchart illustrating a training method for an image reconstruction model according to an embodiment of this application;

[0029] Figure 10 This is a schematic diagram illustrating a process of acquiring training images and reference images according to an embodiment of this application;

[0030] Figure 11 This is a schematic diagram illustrating an offline image reconstruction process according to an embodiment of this application;

[0031] Figure 12 This is a schematic diagram illustrating an online image reconstruction process according to an embodiment of this application;

[0032] Figure 13 This is a flowchart illustrating an image processing method according to an embodiment of this application;

[0033] Figure 14 This is a structural block diagram of an image processing apparatus according to an embodiment of this application;

[0034] Figure 15 This is a structural block diagram of a training device for an image reconstruction model according to an embodiment of this application;

[0035] Figure 16 This is a structural block diagram of another image processing apparatus according to an embodiment of this application;

[0036] Figure 17 This is a structural block diagram of a computing device according to an embodiment of this application;

[0037] Figure 18 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort should fall within the scope of protection of the present application.

[0039] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in other orders. Wherein, "other orders" refers to orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that comprises a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed, or inherent to such processes, methods, products, or apparatus.

[0040] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0041] The technical solution provided in this application is mainly implemented using a deep learning model. Deep learning models can be widely applied in fields such as Natural Language Processing (NLP), computer vision, and speech processing. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and image generation, as well as to natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. Therefore, the main application scenarios of this application include, but are not limited to, digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design. In this application's embodiments, the reconstruction of product images using an image reconstruction model in a product description scenario is used as an example for explanation and illustration.

[0042] According to an embodiment of this application, an image processing method is provided. The steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in a different order than that shown here.

[0043] The technical solutions provided in this application can employ deep learning models with relatively large parameter scales, such as large models containing billions or even more model parameters. Here, "large model" is just one example; this application does not limit the number of model parameters supported by the deep learning model used, aiming to meet actual needs. The deep learning models involved in this application can be artificial intelligence-based language models (LM) or multimodal models (MM).

[0044] Considering the limited computing resources of mobile terminals, the methods described above in this application embodiment can be applied to, for example... Figure 1 The application scenarios shown. Figure 1 This is a schematic diagram illustrating an application scenario of an image processing method according to an embodiment of this application. Figure 1 In the application scenario shown, the deep learning model is deployed on server 10. Server 10 can connect to one or more client devices 20 via a local area network (LAN), wide area network (WAN), internet connection, or other types of data network. Client devices 20 may include, but are not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client devices 20 can interact with users through a graphical user interface to invoke the deep learning model, thereby implementing the method provided in this embodiment.

[0045] In this embodiment, the system consisting of a client device and a server can perform the following steps: the client device acquires the original image; and outputs the target image. The server acquires the initial feature map of the original image; performs attention processing on features of different frequencies in the initial feature map to obtain an enhanced feature map of the original image and an attention weight matrix of the enhanced feature map; adjusts the enhanced feature map based on the attention weight matrix corresponding to the enhanced feature map; and reconstructs the original image based on the spatial feature map to obtain the target image.

[0046] With the rapid development of high-performance computing units, the methods provided in this application embodiment can also be applied to model-in-the-loop machines in other application scenarios. In one optional embodiment, the model-in-the-loop machine has multiple built-in models. Users can select a model to adjust as needed to obtain their own model. The high-performance computing unit built into the model-in-the-loop machine can then directly call the adjusted model to execute the methods provided in this application embodiment. In another optional embodiment, the deep learning model-in-the-loop machine has a pre-trained model built-in. The high-performance computing unit built into the model-in-the-loop machine can then directly call this model to execute the methods provided in this application embodiment.

[0047] Furthermore, when users need to train their own models, they can upload their own datasets via the client. This dataset is sent from the client to the server. The server can then use this dataset to fine-tune the pre-trained model, resulting in the user's customized model, which can then be deployed to the production environment. To facilitate user adjustments, the server provides complete adjustment tools, development frameworks, and processes, supporting various adjustment strategies. This allows the adjusted model to better adapt to different application domains and achieve a high degree of customization.

[0048] Under the aforementioned operating environment, this application provides the following: Figure 2 The image processing method shown. Figure 2 This is a flowchart illustrating an image processing method according to an embodiment of this application. For example... Figure 2 As shown, the method may include the following steps:

[0049] Step S202: Obtain the initial feature map of the original image.

[0050] The initial feature map is used to characterize the visual features of different pixels in the original image.

[0051] The aforementioned original image may refer to an image that needs to be reconstructed to improve the presented visual effect. The aforementioned initial feature map may refer to a feature map constructed based on the visual features of the original image, which can reflect the visual features of different pixels in the original image.

[0052] In scenarios where images are used to display information, such as product introductions, social media displays, online education, telemedicine consultations, and virtual reality experiences, users can often learn about the relevant context by viewing the displayed images. This includes understanding the product's design, the atmosphere of a social event, the content of an educational course, the diagnosis of a medical case, or the immersive experience of a virtual environment. This enhances the user's experience and engagement in various scenarios, enabling them to make more informed and satisfying decisions.

[0053] During image capture, generation, transmission, or display, images may be affected by environmental factors or user viewing needs, leading to quality issues. This can prevent users from clearly identifying the information they need, impacting their experience and decision-making. For example, insufficient device performance or excessively high compression ratios during image capture or transmission can result in images displayed to users exhibiting insufficient resolution, blurred textures, compression artifacts, or jagged edges. Similarly, during image display, if a user needs to examine the texture details of a specific area—such as the knit texture of a shirt's neckline or the interface design of an electronic product—they might zoom in on that area. However, due to the limited resolution and detail of the original image, the zoomed-in image may also suffer from blurring, pixelation, and structural breaks.

[0054] Therefore, in one optional embodiment, to improve the user experience during the actual display of the image to the user, the reconstruction system can first perform quality checks on the image to be displayed to the user. For example, by analyzing the texture coherence, edge sharpness, or compression degree of the image, it can determine whether the image has the aforementioned quality problems. Correspondingly, if the detection result shows no quality problems, the reconstruction system can directly display the image to the user in the graphical user interface, thereby providing the user with a smooth visual experience; while if the detection result shows quality problems, the reconstruction system can determine that the image is the original image that needs to be reconstructed, and improve the adjustment and restoration of the texture and structure of the original image to obtain a target image with higher image quality, thereby significantly improving the user's experience when viewing the image, ensuring that the user can make more accurate decisions based on clear, realistic, and detailed images, and enhancing the user's trust and satisfaction with the platform.

[0055] Furthermore, considering that different users may perceive the visual effects of an image differently—that is, users may have different focuses on different aspects of an image, such as some users prioritizing realism while others may focus more on texture clarity or the integrity of the overall image structure—in addition to the reconstruction system automatically performing quality checks to obtain the original image to be reconstructed, users can also actively select the original image or image region to be reconstructed based on their own observation needs. This provides users with higher-quality target images that better meet their expectations and needs, enhancing their personalized experience and satisfaction.

[0056] Furthermore, considering the process of reconstructing the original image, it can be viewed as restoring and enhancing the image features of the target area with quality problems in the original image to improve the overall clarity, detail richness, and visual realism of the image. The visual features of the original image can well preserve and express the original information and details of the image. Therefore, in order to improve the image quality of the target image obtained by reconstructing the original image, the reconstruction system can start from the visual features of the original image, such as texture, shape, and structure, and combine the correlation between the image features of different pixels to reconstruct the original image. This ensures the coherence and accuracy of restoring the image texture and structure, resulting in a higher quality target image, rather than simply upsampling the original image to obtain a higher resolution image.

[0057] Based on this, during the reconstruction of the original image, the reconstruction system can first extract features from the original image. This is accomplished by a shallow network consisting of a small number of convolutional layers, mapping the original image from the RGB (Red, Green, Blue) pixel space to a high-dimensional feature space. Feature encoding is then performed on the original image to extract low-level visual features such as texture and edges. Next, the reconstruction system can construct an initial feature map based on the extracted low-level visual features. This initial feature map reflects the correlation between global information and local details of the original image, ensuring the accuracy of the reconstruction based on the initial feature map. This guarantees that the generated target image has both high resolution and retains and restores the details and structural features of the original image, thus ensuring a high-quality image reconstruction process and guaranteeing the image quality of the obtained target image.

[0058] The formula for feature extraction from the original image can be:

[0059] .

[0060] in, This represents the extracted shallow visual features. This represents the original image. This represents a shallow network consisting of a small number of convolutional layers.

[0061] Step S204: Attention processing is performed on features of different frequencies in the initial feature map to obtain the enhanced feature map of the original image and the attention weight matrix of the enhanced feature map.

[0062] Among them, the enhanced feature map is used to represent the dependency relationship between different pixels, and the enhanced feature map is constructed based on the attention weight matrix.

[0063] The aforementioned dependencies may include, but are not limited to: the continuity between local pixels in the original image, the texture changes near the object boundary, and the layout associations of the global scene.

[0064] The attention weight matrix described above reflects the correlation and importance between pixels at different locations in the feature map. The attention value of any pixel in the matrix represents its importance within the enhanced feature map—that is, its importance in the overall image reconstruction process. In simpler terms, the attention weight matrix contains the attention weights of different pixels on other pixels within the global view, and can be used to guide how to fuse the feature information from different pixels in the enhanced feature map.

[0065] In one optional embodiment, considering that the dependencies between different pixels are a crucial factor in accurately restoring the texture and structural integrity of the original image during reconstruction, shallow networks have limited ability to understand the high-level semantics of the original image, such as the shape of objects, the layout of scenes, or the properties of materials. The initial feature map constructed using visual features extracted by shallow networks reflects only the shallow visual features of the original image, such as simple textures and edge information. The dependencies between different pixels in the initial feature map are limited to a finite pixel neighborhood or the correlation between shallow visual features, making it difficult to express deeper, more complex dependencies involving a wider range of pixel areas. Therefore, it cannot adequately provide deep-level association information for the reconstruction process, i.e., the aforementioned dependencies. If the original feature map is directly used to restore areas with quality problems in the original image, the restored image may lack textural coherence, affecting the presented visual realism.

[0066] Therefore, in order to improve the accuracy of reconstructing the original image and avoid content distortion, loss of detail, and structural breakage in the reconstructed target image, the reconstruction system can perform frequency division and decoupling processing on the initial feature map after obtaining it. That is, attention processing is applied to features of different frequencies in the initial feature map to identify the association between different key information in the original image, that is, to identify the mutual dependence between different pixels or features in the original image. This enhances the shallow dependence of the initial feature map, resulting in the enhanced feature map mentioned above, as well as the attention weight matrix used in the construction of the enhanced feature map. The attention weight matrix is ​​then used to determine the attention weight of different pixels on other pixels in the global field of view.

[0067] By enhancing feature maps, the reconstruction system can better understand the overall semantics of the original image, while meticulously capturing and preserving the corresponding local details. This ensures that not only the basic texture and structure of the original image can be restored during the reconstruction process, but also the natural transition and coherence of these elements in space can be maintained, thereby generating a high-quality target image that is closer to the real details and structural integrity of the original image.

[0068] The features of different frequencies in the initial feature map mainly reflect different levels of detail and structural characteristics of the original image. Typically, features in the initial feature map can be divided into high-frequency and low-frequency features. High-frequency features focus on local details and edges of the original image, helping the reconstruction system accurately recover the image's microstructure. Low-frequency features focus on the overall structure and global contextual information of the original image, assisting the reconstruction system in understanding the macroscopic framework and semantics of the original image. Correspondingly, by applying attention processing to high-frequency and low-frequency features, the global information and local details of the original image can be effectively integrated with lower computational cost, ensuring the realism and detail richness of the reconstructed target image.

[0069] For ease of understanding, the formula for attention processing of features with different frequencies in the initial feature map can be:

[0070] ;

[0071] .

[0072] in, This represents the query matrix, key matrix, and value matrix of the initial feature map. This represents the initial feature map. This represents the attention processing module. This represents the feature dimension. This represents the activation function.

[0073] Step S206: Adjust the enhanced feature map based on the attention weight matrix to obtain the spatial feature map of the original image.

[0074] Among them, the spatial feature map is used to characterize the semantic information and spatial location correlation between different pixels.

[0075] In one optional embodiment, considering that the reconstruction of the original image typically requires sorting different pixels in the original image to obtain a pixel sequence, and then using a sequence processing model to dynamically update and integrate the features of different pixels according to the sorted pixel sequence, thereby obtaining an adjusted spatial feature representation. The sequence processing model can be a state-space model or a recurrent neural network, and can include multiple processing modules, such as a lightweight state-space module and a parallel convolutional compensation branch. The corresponding sequence model can recursively update the internal state of the enhanced feature map in response to the features of each pixel through the collaborative work of these multiple processing modules, thereby modeling and integrating the dependencies of the entire original image, and thus improving the accuracy of the original image reconstruction.

[0076] While enhanced feature maps can reflect the dependencies between different pixels in the original image, and reconstructing the original image based on enhanced feature maps can improve the accuracy of the reconstructed target image, the directional bias introduced by traditional sorting methods still results in problems such as local texture distortion, structural breaks, or unbalanced information transmission when directly using enhanced feature maps to reconstruct the original image. As a result, the details and structure of the image cannot be fully and accurately restored.

[0077] Specifically, directional bias means that when processing sequences, the model tends to transmit information along a specific path, which may differ from the reality that image details are distributed in any direction. During image reconstruction, local textures or structures may be reconstructed more effectively along this specific path, while reconstruction may be less effective in other directions, leading to overall image distortion or unnatural transitions. Furthermore, due to the fixed sorting method, sequence processing models may not be able to adapt to complex variations and multi-scale dependencies in images, especially when dependencies span large spatial distances. In such cases, the reconstruction capability of sequence processing models will be limited, failing to fully restore the image's details and structural integrity.

[0078] Therefore, in order to further improve the accuracy of reconstructing the original image, the reconstruction system can further construct a spatial feature map that reflects the semantic information and spatial location correlation between different pixels based on the enhanced feature map. The spatial feature map is used to determine the order of processing different pixels in the original image, thereby overcoming directional bias, realizing isotropic texture and structure reconstruction, and ensuring that the order of feature processing is consistent with the inherent semantic correlation of the image content.

[0079] Furthermore, considering that different pixels in the original image play different roles in the reconstruction process, and that the dependencies between them are crucial to the reconstruction quality, especially when the image contains complex textures and structures, the correlation between pixels may span a large spatial distance. Traditional fixed scanning paths often cannot effectively capture such long-distance dependencies, leading to distortion or breaks in certain details and structures in the reconstructed image. Therefore, during the construction of the spatial feature map of the original image, the reconstruction system can use the attention weight matrix corresponding to the enhanced feature map to adjust the enhanced feature map, thereby achieving content-driven dynamic scanning. This prioritizes processing pixels with high semantic importance and rich information, dynamically adjusting the pixel processing order to eliminate the influence of directional bias, while preserving and enhancing key features, thus obtaining the spatial feature map of the original image.

[0080] Step S208: Reconstruct the original image based on the spatial feature map to obtain the target image.

[0081] In one optional embodiment, after obtaining the spatial feature map, the reconstruction system can use the spatial feature map to reconstruct the original image to obtain the corresponding target image. For example, the reconstruction system can perform feature upsampling and image space mapping based on the spatial feature map to restore the details and structure of the high-resolution image. The upsampled feature map can be fed into a reconstruction network to achieve mapping from the high-dimensional feature space to the low-dimensional RGB image space. The reconstruction network generates the target image based on the information in the upsampled feature map through a combination of a series of convolutional layers and activation functions. In this process, the role of the reconstruction network is to restore the texture and structural details of the image while maintaining the global semantic information of the image, ensuring that the target image has both high-resolution clarity and preserves the content integrity of the original image.

[0082] In this embodiment, the method involves obtaining an initial feature map of the original image; performing attention processing on features of different frequencies in the initial feature map to obtain an enhanced feature map of the original image and an attention weight matrix for the enhanced feature map; adjusting the enhanced feature map based on the attention weight matrix to obtain a spatial feature map of the original image; and reconstructing the original image based on the spatial feature map to obtain the target image. By performing attention processing on the initial feature map of the original image and then adjusting the enhanced feature map obtained during the attention processing using the attention weight matrix to obtain the spatial feature map, the dependency between different pixels and the correlation between semantic information and spatial location between different pixels can be effectively enhanced. This improves the accuracy of reconstructing the original image using the spatial feature map and effectively improves the image quality of the reconstructed target image, thereby solving the technical problem of poor image quality in related technologies.

[0083] In this embodiment, the attention weight matrix includes a first weight matrix and a second weight matrix. Attention processing is performed on features of different frequencies in the initial feature map to obtain an enhanced feature map of the original image. This includes: inputting the initial feature map into a first decoupling module, performing attention processing on the initial feature map using the first decoupling module to obtain a first feature and a first weight matrix of the original image; inputting the initial feature map into a second decoupling module, performing attention processing on the initial feature map using the second decoupling module to obtain a second feature and a second weight matrix of the original image, wherein the frequency corresponding to the first feature in the spatial frequency domain is greater than the frequency corresponding to the second feature in the spatial frequency domain; and fusing the first feature and the second feature to obtain the enhanced feature map.

[0084] The first feature mentioned above can refer to high-frequency features, including but not limited to features such as edges, local textures, and local brightness variations. These features typically carry subtle details and complex structures in the image, affecting the visual sharpness and realism of the original image. The second feature mentioned above can refer to low-frequency features, including but not limited to features such as overall contours, object regions, and background element distribution. These features are responsible for the basic structure and macroscopic information of the image, affecting the reconstruction team's structural integrity and semantic understanding of the original image. Correspondingly, the frequency of the first feature in the spatial frequency domain is usually greater than the frequency of the second feature in the spatial frequency domain. Here, the frequency in the spatial frequency domain refers to the scale and wavelength of the feature in the image. High-frequency features correspond to short wavelengths and small-scale changes, while low-frequency features correspond to long wavelengths and large-scale information.

[0085] In one optional embodiment, in order to achieve efficient and targeted processing of different frequency features of the image, during the attention processing of different frequency features in the initial feature map, the reconstruction system can first perform attention processing on the initial feature map to identify the first and second features of different frequencies, as well as the first weight matrix corresponding to the first feature and the second weight matrix corresponding to the second feature from the initial feature map, and then construct the enhanced feature map of the original image based on the first and second features. To avoid confusion and improve the accuracy of the extracted first and second features, during the attention processing of the initial feature map, corresponding first and second decoupling modules can be set for the first and second features, respectively. The first decoupling module is used to perform attention processing on the initial feature map, preserving and enhancing high-frequency features such as local textures and edges in the original image, thereby obtaining a more accurate and complete first feature and first weight matrix of the original image. The second decoupling module is used to perform attention processing on the initial feature map, capturing low-frequency features such as the overall structure and semantic context of the original image, thereby obtaining a more accurate and complete second feature and second weight matrix of the original image. This ensures the detail clarity of high-frequency features and the structural stability of low-frequency features, realizing frequency band decoupling and independent adjustment of features. Finally, an enhanced feature map that balances detail and structural integrity is obtained through feature fusion.

[0086] In this embodiment, an initial feature map is input to a first decoupling module, and attention processing is performed on the initial feature map using the first decoupling module to obtain a first feature and a first weight matrix of the original image. This includes: performing convolution processing on the initial feature map based on first convolution parameters to obtain a first query matrix, a first key matrix, and a first value matrix of the initial feature map; obtaining a first weight matrix based on the dot product of the first query matrix and the first key matrix; upsampling the first weight matrix based on the resolution corresponding to the initial feature map to obtain an upsampled matrix; weighting the upsampled matrix and the first value matrix to obtain a first weighted matrix; and performing convolution processing on the first weighted matrix to obtain a first feature.

[0087] In one optional embodiment, to improve the accuracy of the extracted first feature, during the attention processing of the initial feature map using the first decoupling module, the reconstruction system can first perform convolution processing on the initial feature map according to the first convolution parameters to obtain the first query matrix, the first key matrix, and the first value matrix of the initial feature map. Then, based on the dot product of the first query matrix and the first key matrix, the first weight matrix is ​​obtained. The first weight matrix is ​​then upsampled according to the resolution corresponding to the initial feature map to obtain the upsampled matrix. Finally, the upsampled matrix and the first value matrix are weighted to obtain the first weighted matrix, and the first weighted matrix is ​​convolved to obtain the first feature.

[0088] For ease of understanding, the formula for extracting the first feature using the first decoupling module may include:

[0089] ;

[0090] ;

[0091] ;

[0092] .

[0093] in, This represents the first query matrix. This represents the first key matrix. This represents the first-valued matrix. This represents the first weight matrix. This represents the upsampling matrix. This represents the primary characteristic.

[0094] Figure 3 This is a schematic diagram illustrating a first decoupling module according to an embodiment of this application. For example... Figure 3 As shown, in the first decoupling module, the reconstruction system first uses a convolutional layer with a stride of S to convolve the initial feature map, obtaining a first query matrix and a first key matrix. Simultaneously, it uses a convolutional layer with a stride of 1 to convolve the initial feature map, obtaining a first value matrix. Then, it performs a dot product operation on the first query matrix and the first key matrix to obtain a first weight matrix. Next, the reconstruction system upsamples the first weight matrix to obtain an upsampled matrix, and then performs a weighted operation on the upsampled matrix and the first value matrix. Finally, it performs a convolution operation on the weighted first weighted matrix to obtain the aforementioned first feature.

[0095] In this embodiment, an initial feature map is input to a second decoupling module, and attention processing is performed on the initial feature map using the second decoupling module to obtain a second feature and a second weight matrix of the original image. This includes: pooling the initial feature map to obtain a pooled feature map; convolving the initial feature map based on second convolution parameters to obtain a second query matrix and a second key matrix of the initial feature map, and convolving the initial feature map based on third convolution parameters to obtain a second value matrix of the initial feature map; obtaining a second weight matrix based on the dot product of the second query matrix and the second key matrix; weighting the second weight matrix and the second value matrix to obtain a second weighted matrix; and convolving the second weighted matrix to obtain the second feature.

[0096] In one optional embodiment, to improve the accuracy of the extracted second feature, during the attention processing of the initial feature map using the second decoupling module, the reconstruction system can first perform pooling processing on the initial feature map to obtain a pooled feature map, and then perform convolution processing on the initial feature map using the second convolution parameters to obtain the second query matrix and the second key matrix of the initial feature map. Then, the reconstruction system can perform convolution processing on the initial feature map according to the third convolution parameters to obtain the second value matrix of the initial feature map, and obtain the second weight matrix based on the dot product of the second query matrix and the second key matrix. Finally, the second weight matrix and the second value matrix are weighted to obtain the second weighted matrix, and the second weighted matrix is ​​convolved to obtain the second feature.

[0097] For ease of understanding, the formula for extracting the first feature using the first decoupling module may include:

[0098] ;

[0099] ;

[0100] ;

[0101] ;

[0102] .

[0103] in, This represents the second query matrix. This represents the second bond matrix. This represents a second-valued matrix. This represents the second weight matrix. This represents the second characteristic. This represents the pooling feature map.

[0104] Figure 4 This is a schematic diagram illustrating a second decoupling module according to an embodiment of this application. Wherein, as... Figure 4 As shown, in the second decoupling module, the reconstruction system first uses a pooling layer with a stride of S to pool the initial feature map, obtaining the pooled feature map mentioned above. Then, it uses a convolutional layer with a stride of 1 to convolve the initial feature map and the pooled feature map respectively, obtaining the second query matrix, the second key matrix, and the second value matrix of the initial feature map. Next, the reconstruction system performs a dot product operation on the second query matrix and the second key matrix to obtain the second weight matrix. Finally, the reconstruction system performs a weighted average operation on the second weight matrix and the second value matrix, and then performs a convolution operation on the resulting second weighted matrix to obtain the second feature map mentioned above.

[0105] Correspondingly, the formula for fusing the first feature and the second feature can be:

[0106] .

[0107] in, This represents the enhanced features obtained through fusion. By utilizing the enhanced features corresponding to different elements, an enhanced feature map corresponding to the original image can be constructed.

[0108] In this embodiment of the application, the spatial feature map of the original image is obtained by adjusting the enhanced feature map based on the attention weight matrix, including: performing dimensional transformation on the enhanced feature map based on the first weight matrix to obtain an initial feature sequence, wherein the initial feature sequence is used to characterize the processing order of features corresponding to different pixels in the enhanced feature map; reconstructing the initial feature sequence based on the second weight matrix to obtain a target feature sequence; and reshaping the target feature sequence based on the resolution of the enhanced feature map to obtain a spatial feature map.

[0109] Dimensional transformation can refer to flattening the enhanced feature map, converting it from a two-dimensional image into a one-dimensional sequence.

[0110] The formula for dimension transformation can be:

[0111] .

[0112] in, This represents the one-dimensional sequence obtained after the transformation. It is the sequence length.

[0113] In one optional embodiment, considering that the first weight matrix can reflect the correlation between different pixels on high-frequency features, the reconstruction system can sort the pixels in the enhanced feature map according to the first weight matrix to obtain an initial feature sequence that reflects the processing order of the corresponding features of different pixels in the enhanced feature map. However, considering that there may be some redundant low-response pixels in the enhanced feature map, directly using the initial feature sequence as input to the state space model may lead to wasted computational resources and low processing efficiency. Therefore, after obtaining the initial feature sequence, the reconstruction system can reconstruct the initial feature sequence to remove redundant pixels with low information content, thereby improving the utilization of computational resources and processing efficiency. Furthermore, considering that the second weight matrix can reflect the low-frequency global contextual correlation between different pixels, the reconstruction system can reconstruct the initial feature sequence according to the second weight matrix to obtain the aforementioned target feature sequence, thereby ensuring that when processing long-distance dependencies, global structure and semantic information are fully considered, avoiding global incoherence caused by local adjustments. Finally, after obtaining the target feature sequence, the reconstruction system can reshape the target feature sequence using the resolution of the enhanced feature map, that is, convert the target feature sequence back to a two-dimensional spatial layout with the same resolution as the enhanced feature map, and obtain the aforementioned spatial feature map.

[0114] In this embodiment, the enhanced feature map is dimensionally transformed based on the first weight matrix to obtain an initial feature sequence, including: upsampling the first weight matrix based on the resolution corresponding to the initial feature map to obtain a third weight matrix; determining the first semantic saliency score and symmetric affinity matrix corresponding to any pixel based on the third weight matrix, wherein the first semantic saliency score is used to characterize the correlation between the pixel and other pixels, and the symmetric affinity matrix is ​​used to characterize the similarity between the pixel and other pixels; determining the pixel processing order of multiple pixels based on the first semantic saliency score and symmetric affinity matrix corresponding to multiple pixels in the enhanced feature map; and performing dimensional transformation on the enhanced feature map based on the pixel processing order to construct the initial feature sequence.

[0115] In one optional scheme of this embodiment, considering that the first weight matrix is ​​obtained by processing the first query matrix Q1 and the first key matrix K1, and the first query matrix Q1 and the first key matrix K1 are obtained by processing the initial feature map using a convolutional layer with a stride of S, the spatial resolution of the first query matrix Q1 and the first key matrix K1 will be reduced from the original H and W to H / S and W / S. The resolution of the corresponding first weight matrix will be smaller than the resolution of the enhanced feature map. Blindly using the first weight matrix to perform dimensional transformation on the enhanced feature map may lead to inaccurate pixel correlation evaluation due to the difference in resolution, and anisotropic deviation that may be introduced by the fixed scanning path, resulting in problems such as reduced information matching degree and directional deviation.

[0116] Therefore, in order to improve the rationality and accuracy of the obtained initial feature sequence, during the dimensionality transformation of the enhanced feature map using the first weight matrix, the reconstruction system can first use the resolution corresponding to the initial feature map to upsample the first weight matrix to obtain the aforementioned third weight matrix. Then, the third weight matrix is ​​used to determine the first semantic saliency score and symmetric affinity matrix corresponding to any pixel in the enhanced feature map. The first saliency score is used to determine the similarity between the pixel and other pixels, ensuring that the reconstruction system can identify which pixels carry more local detail information, thus prioritizing the processing of these information-dense pixels. The symmetric affinity matrix is ​​used to determine the similarity between the pixel and other pixels, thus ensuring that the reconstruction system can eliminate possible anisotropic biases based on the symmetric affinity matrix, and ensuring that the similarity evaluation from each pixel to other pixels is bidirectional.

[0117] By using the first semantic saliency score and symmetric affinity matrix corresponding to different pixels, the reconstruction system can determine the processing order between different pixels, i.e., the pixel processing order mentioned above. Finally, based on the determined pixel processing order, the reconstruction system can perform dimensionality transformation on the enhanced feature map to construct the initial feature sequence mentioned above.

[0118] The formula for constructing the initial feature sequence may include:

[0119] ;

[0120] ;

[0121] ;

[0122] ;

[0123] .

[0124] ;

[0125] .

[0126] in, This represents the third weight matrix. This represents the first semantic affinity matrix. This represents the symmetric affinity matrix. It represents multiple cluster sets. This represents the importance index of the k-th cluster set. This represents the processing order of pixels corresponding to the k-th cluster set. This represents the initial feature sequence.

[0127] In this embodiment, the pixel processing order of multiple pixels is determined based on the first semantic saliency score and the symmetric affinity matrix corresponding to multiple pixels in the enhanced feature map. This includes: clustering multiple pixels based on the symmetric affinity matrix to obtain multiple cluster sets; determining an importance index for each cluster set based on the first semantic saliency score of at least one pixel contained in any cluster set, wherein the importance index characterizes the importance of pixels in the cluster set during image reconstruction; sorting the multiple cluster sets based on the importance index to obtain a processing order for the multiple cluster sets; and sorting the multiple pixels based on the processing order and the first semantic saliency score of any pixel contained in any cluster set to obtain a pixel processing order.

[0128] In one optional embodiment, considering the strong inherent correlation and information complementarity among pixels with high similarity during processing, they should be processed simultaneously or consecutively to maintain the coherence of texture and structure. Therefore, to improve the rationality of the determined pixel processing order, the reconstruction system can first cluster multiple pixels using a symmetric affinity matrix to obtain multiple cluster sets. Then, based on the first semantic saliency score of at least one pixel contained in any cluster set, the importance index of the corresponding cluster set is determined. This importance index determines the degree of importance of the pixels in this cluster set in the image reconstruction process. The higher the importance index, the higher the corresponding importance. The pixels in this cluster set play a more crucial role in the detail reconstruction and structural restoration of the image. In this case, the reconstruction system can first process the pixels contained in the cluster set with the higher importance index to prioritize the restoration of key details and structural information in the image, thereby more effectively utilizing limited computing resources and improving the overall efficiency and quality of image reconstruction.

[0129] Based on this, after obtaining the importance index corresponding to different cluster sets, the reconstruction system can use the importance index to sort multiple cluster sets, obtain the processing order of different cluster sets, and then in any cluster set processing order, sort different pixels according to the first semantic saliency score corresponding to the pixels contained in that cluster set, thereby obtaining the above-mentioned pixel processing order.

[0130] In this embodiment of the application, the initial feature sequence is reconstructed based on the second weight matrix to obtain the target feature sequence, including: determining the second semantic saliency score of any pixel in the initial feature sequence based on the second weight matrix; selecting multiple target pixels from the multiple pixels contained in the initial feature sequence based on the second semantic saliency score to obtain the target feature sequence, wherein the second semantic saliency score corresponding to any target pixel is greater than a preset score threshold.

[0131] In one optional embodiment, to reduce the waste of computational resources, during the reconstruction of the initial feature sequence based on the second weight matrix, the reconstruction system can first determine the second semantic saliency score of any pixel in the initial feature sequence based on the second weight matrix. Then, based on the second semantic saliency score, it selects multiple target pixels from the multiple pixels contained in the initial feature sequence to obtain the target feature sequence. This ensures that the reconstruction system can process only pixels carrying key global information, achieving efficient utilization of computational resources and accurate modeling of the global structure of the image. The difference between the second semantic saliency score and the first semantic saliency score lies in their calculation method and the type of information they emphasize. The first semantic saliency score is calculated based on the attention weight matrix of the high-frequency detail branch, focusing on the details of local texture and edges in the image. The second semantic saliency score, on the other hand, is based on the attention weight matrix of the low-frequency global branch, focusing more on the macroscopic structure of the image and low-frequency contextual information. The second semantic saliency score of any target pixel selected from multiple pixels is greater than a preset score threshold. This ensures that the input sequence of the state space model contains only elements that are crucial to the global structure reconstruction of the image, avoids redundant calculations, improves the processing speed and resource utilization of the model, and ensures the integrity and reconstruction quality of the key structures of the image.

[0132] The formula for constructing the second semantic saliency score can be:

[0133] .

[0134] The selected target feature sequence can be represented as:

[0135] .

[0136] Where N is less than HW.

[0137] In this embodiment, reshaping the target feature sequence to obtain a spatial feature map includes: inputting the target feature sequence into a state space model, updating the hidden state of the target feature sequence using the state space model to obtain an updated feature sequence; reshaping the updated feature sequence based on a first operator to obtain a first feature map, and reshaping the initial feature sequence based on a second operator to obtain a second feature map, wherein the resolution of the first feature map, the resolution of the second feature map, and the resolution of the enhanced feature map are the same; inputting the second feature map into a parallel convolution model, updating the second feature map using the parallel convolution model to obtain a third feature map; and fusing the first feature map and the third feature map to obtain a spatial feature map.

[0138] In one optional embodiment, to improve the accuracy of reconstructing the target feature sequence, the reconstruction system can first input the obtained target feature sequence into a state-space model, update the hidden state of the target feature sequence using the state-space model to obtain the updated feature sequence, and then reconstruct the updated feature sequence using a preset first operator to obtain a first feature map. The state-space model can refer to a lightweight recurrent neural network based on discrete state equations. The state-space model can include multiple processing modules, such as a state transition module responsible for updating the model's hidden state based on the current input and previous states; an output mapping module used to map the updated hidden state to the model's output; and an intelligent skipping mechanism that dynamically selects the input sequence based on the semantic importance of pixels to improve computational efficiency. The state-space model can utilize these processing modules to update the hidden state of the target feature sequence to enhance the long-term dependency expression ability of image features and the preservation of local details, thereby improving the expressive richness and information integrity of the obtained updated feature sequence.

[0139] The formula for updating the hidden state can be:

[0140] ;

[0141] .

[0142] in, This represents the hidden state. This represents the k-th element of the input sequence. and This represents the state transition matrix after zero-order preserved discretization. and This represents the output mapping matrix.

[0143] Furthermore, considering that the target feature sequence is a feature sequence obtained by filtering multiple pixels contained in the initial feature sequence, the number of pixels contained in the target feature sequence is usually less than the number of feature points contained in the enhanced feature map. To ensure pixel alignment accuracy and feature map integrity during the reshaping process, the reconstruction system needs to maintain the resolution of the reshaped first feature map at the same level as the enhanced feature map. Correspondingly, during the reshaping of the target feature vector, the reconstruction system needs to fill in the pixels not selected by the intelligent skipping mechanism to maintain the consistency of the feature map's size and structure. To avoid potential information distortion introduced by filling, a preset feature value, such as 0, is usually used to directly fill the blank positions in the first feature map. However, this simple zero-filling method may destroy the local coherence and semantic consistency of the feature map. Correspondingly, if the reshaped first feature map is directly determined as the spatial feature map, local detail loss or structural breakage may occur, leading to a decrease in the visual quality of the image reconstruction.

[0144] Therefore, to ensure that the resolution of the reconstructed spatial feature map is the same as that of the enhanced feature map, and to avoid blindly filling pixels in the first feature map that are not covered by the target feature sequence, which would lead to distortion of image feature representation and reduced reconstruction effect, the reconstruction system can also use a second operator to reconstruct the initial feature sequence to obtain a second feature map with the same resolution as the enhanced feature map. Then, the feature values ​​corresponding to different pixels in the second feature map are used to fill the feature values ​​of the pixels in the first feature map that need to be filled. Considering that the second feature map retains the complete information of all pixels, including the features of low-response or redundant regions excluded in the target feature sequence, when filling the first feature map with the feature values ​​corresponding to different pixels in the second feature map, the reconstruction system can also first input the second feature map into a parallel convolution model, use the parallel convolution model to update the second feature map to obtain the aforementioned third feature map, and then fuse the first and third feature maps to obtain the corresponding spatial feature map, thereby achieving the fusion of global structure and local details, improving the quality and visual effect of image reconstruction.

[0145] The parallel convolutional model can include multiple processing modules, such as depthwise separable convolutional modules and attention modules. By updating the second feature map through these multiple processing modules, the local texture and edge details of the image can be further enhanced. At the same time, through the channel-level attention mechanism of the attention module, the weights of different feature channels are adjusted to highlight the key details of the image, thereby adjusting the filling effect of blank positions in the first feature map and ensuring the integrity, coherence and visual quality of the overall feature map.

[0146] The processing formula for parallel convolution models can be:

[0147] .

[0148] Based on this, by introducing a parallel convolutional model, the second feature map can not only supplement the pixel information excluded by the intelligent skipping mechanism, but also further adjust the feature expression on this basis, providing more accurate and richer feature values ​​for filling the first feature map. This ensures that the spatial feature map has more delicate and realistic local detail information on the basis of size consistency, and finally achieves the balance and coordination between global structure and local details in image reconstruction, which greatly improves the accuracy and visual effect of image super-resolution reconstruction.

[0149] Correspondingly, the formula for fusing the first and third features can be expressed as:

[0150] .

[0151] In this embodiment, after adjusting the enhanced feature map based on the attention weight matrix to obtain the spatial feature map of the original image, the method further includes: obtaining the number of times the spatial feature map is constructed; evaluating the spatial feature map to obtain a feature map evaluation result, wherein the feature map evaluation result is used to characterize the accuracy of the spatial feature map; if the number of constructions is less than a preset threshold, or the feature map evaluation result characterizes the accuracy of the spatial feature map less than a preset accuracy threshold, the spatial feature map is determined as a new initial feature map, and the frequency division and decoupling of the initial feature map is re-executed to construct the enhanced feature map of the original image; the enhanced feature map is adjusted based on the attention weight matrix to obtain the spatial feature map of the original image, until the feature map evaluation result corresponding to the spatial feature map characterizes the accuracy of the spatial feature map to be greater than or equal to a preset accuracy threshold, or the number of constructions is greater than or equal to a preset threshold.

[0152] In one optional embodiment, considering that a single adjustment process may not be sufficient to adequately adjust the feature representation and achieve the image reconstruction effect expected by the user, after adjusting the enhanced feature map using the attention weight matrix, the reconstruction system can also obtain the number of times the spatial feature map has been constructed, i.e., the number of times the feature extraction and attention adjustment processes have been iterated, to control the duration and number of model iterations. Simultaneously, the reconstruction system can also evaluate the currently obtained spatial feature map to obtain a feature map evaluation result, which reflects the accuracy of the obtained spatial feature map. For example, during the evaluation of the spatial feature map, the reconstruction system can compare the pixel-level differences between the spatial feature map and the original high-resolution image, as well as the high-level structural similarity, to obtain a comprehensive evaluation index.

[0153] After obtaining the number of construction attempts and feature map evaluation results, if the number of construction attempts is less than the preset number, or the feature map evaluation results show that the accuracy of the spatial feature map is less than the preset accuracy threshold, it indicates that the currently constructed spatial map may have missing details, local distortion, or other defects. If the original image is directly reconstructed based on the current spatial feature map, it may result in poor image quality that cannot meet application requirements. Therefore, if the number of construction attempts is less than the preset number, or the feature map evaluation results show that the accuracy of the spatial feature map is less than the preset accuracy threshold, the reconstruction system can re-execute the process of constructing the spatial feature map. That is, the spatial feature map is determined as the new initial feature map, and the above-mentioned frequency division and decoupling of the initial feature map is re-executed to construct the enhanced feature map of the original image. The enhanced feature map is then adjusted based on the attention weight matrix to obtain the spatial feature map of the original image. This process continues until the feature map evaluation result corresponding to the spatial feature map indicates that the accuracy of the spatial feature map is greater than or equal to the preset accuracy threshold, or the number of construction attempts is greater than or equal to the preset number of attempts threshold.

[0154] By iterating in this way, the model's ability to extract and represent image features can be continuously adjusted, ensuring that the final spatial feature map meets the requirements of high-quality image reconstruction, while controlling the model training time and resource consumption within a reasonable range.

[0155] In this embodiment of the application, reconstructing the original image based on the spatial feature map to obtain the target image includes: inputting the spatial feature map into an upsampling module, using the upsampling module to perform upsampling processing on the spatial feature map to obtain an upsampled feature map, wherein the resolution of the upsampled feature map is greater than the resolution of the spatial feature map; and reconstructing the original image based on the upsampled feature map to obtain the target image.

[0156] In one optional embodiment, during the reconstruction of the original image, to ensure the richness of detail and visual quality of the reconstructed image while keeping computational efficiency and resource consumption within a controllable range, the reconstruction system can first input the spatial feature map into the upsampling module. The upsampling module then performs upsampling processing on the spatial feature map to obtain an upsampled feature map with a resolution greater than that of the original spatial feature map. This expands the feature map size while maintaining detail integrity, providing sufficient information density and spatial resolution for subsequent image reconstruction. Then, the reconstruction system can further reconstruct the original image based on the upsampled feature map to obtain a high-fidelity, artifact-free, and detail-rich target image.

[0157] The formula for upsampling the spatial feature map can be:

[0158] .

[0159] in, This represents a spatial feature map. This represents the upsampled feature map.

[0160] The formula for image reconstruction based on upsampled feature maps can be:

[0161] .

[0162] To facilitate understanding of the reconstruction process described above, Figure 5 This is a schematic diagram illustrating an image reconstruction process according to an embodiment of this application. Figure 5 As shown, in the process of reconstructing the original image, the reconstruction system can first perform convolution processing on the original image to extract the shallow visual features of the original image to construct the corresponding initial feature map. Then, the initial feature map is input into multiple sequentially connected frequency band decoupled dynamic state space blocks. The dynamic state space blocks are used to process the initial feature map to obtain the corresponding spatial feature map. Finally, the spatial feature map is upsampled and convolved to obtain a target image with higher resolution.

[0163] In any dynamic state space block, the reconstruction system first decouples the initial feature map for high and low frequencies. This involves using high-frequency and low-frequency branches to perform attention processing on the initial feature map, extracting high-frequency and low-frequency features from the original image, and then fusing these features to obtain an enhanced feature map. After obtaining the enhanced feature map, the reconstruction system dynamically scans it to determine the order in which features corresponding to different pixels in the original image are processed, resulting in a target feature sequence. Following this target feature sequence, the reconstruction system performs lightweight processing to reduce computational complexity, accelerate model execution, and maintain the reconstruction quality of the global structure and local texture of the image. This yields the spatial feature map output by the initial feature map within that dynamic state space block.

[0164] In the high- and low-frequency decoupling process, the reconstruction system utilizes a high-frequency branch to calculate the Q and K matrices through a convolutional layer with a stride of S, reducing the spatial resolution of the initial feature map while maintaining the original resolution of the V matrix. This enables efficient querying and processing of local information. The low-frequency branch applies adaptive average pooling to the input features to construct a compact global summary for contextual queries. This reduces the spatial size of the feature map before calculating K and V, achieving a combination of full-resolution querying and global summary response.

[0165] During dynamic scanning, the reconstruction system first reads the high-frequency attention weight matrix and upsamples it to match the resolution of the initial feature map. Then, it calculates the semantic saliency score for each spatial location. Based on the scores, a symmetric affinity matrix is ​​constructed, and spectral clustering is performed to obtain K semantic clusters, each corresponding to different semantic content. Next, the importance of the clusters is calculated, and all clusters are sorted. They are then sorted again based on the saliency scores of pixels within each cluster, ultimately determining a rearranged sequence order π, which is the initial feature sequence. This dynamic scanning method overcomes the modeling bias caused by the fixed scanning path in traditional state-space models, improving the rationality of the constructed target feature sequence.

[0166] During lightweight processing, the reconstruction system can further simplify the sequence based on the dynamically scanned and determined sequence using an intelligent skipping mechanism, retaining only the features corresponding to semantically significant pixels to generate the target feature sequence. This significantly reduces the computational load of the state-space model and improves processing efficiency. Simultaneously, by introducing parallel convolutional branches to supplement the local details lost due to intelligent skipping, a balance between global and local modeling can be achieved.

[0167] Figure 6 This is a schematic diagram illustrating a comparison of scan results according to an embodiment of this application. Figure 6 The top side shows the scanning results using the traditional scanning method, and the bottom side shows the scanning results using the dynamic scanning method described in this application. For example... Figure 6 As shown, traditional scanning methods typically determine the processing order of features corresponding to each pixel in the original image based on preset row-first, column-first, or zigzag scanning rules. This can lead to directional deviations due to the fixed scanning path, resulting in a model that is stronger in some directions but weaker in directions perpendicular to the scanning path. For example, when processing an image of horizontal stripes, using a row-first scanning method might result in good modeling of the stripe details, but insufficient processing of other structures perpendicular to the stripes.

[0168] The dynamic scanning method mentioned in this application determines the processing order of features corresponding to each pixel in the original image based on the semantic saliency score calculated by the high-frequency attention weight matrix. In contrast, the dynamic scanning method of this application ensures that the model can extract information based on the semantic and structural relevance of the content when processing images, overcoming the directional bias of traditional scanning methods and improving the model's understanding and modeling ability of the global context and local details of the image. The dynamic scanning method allows the model to prioritize semantically important regions when processing images, such as the texture of clothing and the contour of faces, thereby achieving more efficient and accurate feature extraction and ensuring the visual quality and detail recovery ability of the reconstructed image. In addition, the dynamic scanning mechanism can automatically skip feature extraction of low-response and redundant regions through an intelligent skip strategy, further reducing computational costs. At the same time, it compensates for the local information loss caused by skip sampling through parallel convolutional branches, achieving a balance between global structure and local details.

[0169] To visually demonstrate the effect of the above method on reconstructing the original image, Figure 7 This is a schematic diagram illustrating a comparison of reconstruction results according to an embodiment of this application. Figure 8 This is a schematic diagram illustrating another reconstruction effect comparison result according to an embodiment of this application. Wherein, as... Figure 7 Taking the area around the person's neck where the necklace is located as an example, Figure 7 The image on the left represents a magnified view of the area where the necklace is located. Figure 7 The image on the right represents the reconstructed image of the area where the necklace is located, obtained using the image reconstruction method described above. For example... Figure 8 As shown, taking the enlarged area of ​​the book as an example, Figure 8 The image on the left represents a magnified view of the area where the book is located. Figure 8 The image on the right represents the image obtained after reconstructing the area where the book is located using the image reconstruction method described above.

[0170] pass Figure 7 and Figure 8 Comparing the visual effect of a normally magnified image with that of an image reconstructed using the aforementioned image reconstruction method, it can be seen that the normally magnified image appears blurry, while the image reconstructed using the aforementioned method is clearer. This method can accurately and effectively reconstruct the image while restoring its original structure and texture, thereby improving the quality of image reconstruction.

[0171] For the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions. This is because, according to this application, certain steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms, or by hardware. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0173] According to embodiments of this application, a method for training an image reconstruction model is also provided. Figure 9 This is a flowchart illustrating a training method for an image reconstruction model according to an embodiment of this application, such as... Figure 9 As shown, the method may include the following steps:

[0174] Step S902: Obtain the training image and the reference image.

[0175] Step S904: Input the training image into the initial reconstruction model, and use the initial reconstruction model to reconstruct the training image to obtain the reconstructed image.

[0176] Step S906: Based on the reconstructed image and the reference image, construct the model loss function of the initial reconstruction model.

[0177] Step S908: Adjust the initial reconstruction model using the model loss function to obtain the image reconstruction model.

[0178] The image reconstruction model is used to perform the image processing methods described above.

[0179] The training images mentioned above refer to low-resolution input images, while the reference images are high-resolution images.

[0180] In one optional embodiment, to improve the accuracy of reconstructing the original image, the aforementioned image reconstruction model can be configured in the reconstruction system to execute the image processing method described above. Correspondingly, during the training of the image reconstruction model, the reconstruction system can first acquire the aforementioned training image and reference image, and then input the training image into the initial reconstruction model. The initial reconstruction model is then used to reconstruct the training image to obtain the reconstructed image. The reconstruction system constructs the model loss function for the initial reconstruction model by building the reconstructed image and reference image, and adjusts the initial reconstruction model using the model loss function to obtain the image reconstruction model.

[0181] In this embodiment of the application, obtaining training images and reference images includes: obtaining sample images; performing image degradation on the sample images to obtain training images; upsampling the sample images to obtain upsampled images, and performing image enhancement on the sample images to obtain enhanced images; and fusing the upsampled images and enhanced images to obtain reference images.

[0182] In one optional embodiment, considering that in the traditional training process of image reconstruction models, in order to simulate the degradation process of real images, a degradation model is usually designed to apply strategies such as geometric degradation, blur degradation, compression degradation, and noise degradation to the sample images in sequence, so that the generated training images are consistent with the real low-quality images in terms of statistical characteristics, and then the sample images are used as reference images for training. However, the upper limit of the reconstruction capability of the image reconstruction model trained in this way depends on the image quality of the sample images. If the sample images themselves are of poor quality, then the trained image reconstruction model will have difficulty reconstructing high-quality images.

[0183] Based on this, in order to improve the capabilities of the trained image reconstruction model, it is possible to follow... Figure 10 The process of constructing training and reference images is shown, and the aforementioned training and reference images are obtained. Figure 10 This is a schematic diagram illustrating a process for acquiring training images and reference images according to an embodiment of this application, such as... Figure 10 As shown, in the process of acquiring training and reference images, the reconstruction system can acquire sample images, perform image degradation on the sample images to obtain training images, then upsample the sample images to obtain upsampled images, perform image enhancement on the sample images to obtain enhanced images, and finally fuse the upsampled images and enhanced images to obtain reference images.

[0184] In this embodiment of the application, a model loss function for an initial reconstruction model is constructed based on the reconstructed image and the reference image, including: obtaining pixel loss, perceptual loss, relative discriminative adversarial loss and artifact suppression loss between the reconstructed image and the reference image, wherein the relative discriminative adversarial loss is used to characterize the relative realism between the reconstructed image and the reference image, and the artifact suppression loss is used to characterize the degree of artifacts appearing in the reconstructed image; the pixel loss, perceptual loss, relative discriminative adversarial loss and artifact suppression loss are weighted to obtain the model loss function.

[0185] In one optional embodiment, to improve the accuracy of adjusting the initial reconstruction model using the model loss function, the reconstruction system can first acquire the pixel loss, perceptual loss, relative discriminative adversarial loss, and artifact suppression loss between the reconstructed image and the reference image when constructing the model loss function. The pixel loss reflects the differences between the reconstructed image and the reference image at the pixel level, ensuring that the reconstructed image is as close as possible to the target image numerically. The perceptual loss reflects the similarity between the reconstructed image and the reference image at the perceptual level. The discriminative adversarial loss reflects the realism of the reconstructed image relative to the reference image in the eyes of the discriminator. The artifact suppression loss reflects the self-consistency and stability of the reconstructed image in high-frequency detail processing, ensuring that common high-frequency artifacts are not generated during image super-resolution. Then, the pixel loss, perceptual loss, relative discriminative adversarial loss, and artifact suppression loss are weighted to construct the aforementioned model loss function, thereby achieving a comprehensive balance between pixel accuracy, visual perceptual similarity, image realism, and output stability.

[0186] The formula for the model loss function can be:

[0187] .

[0188] in, This represents the model loss function. This represents pixel loss. This represents perceived loss. This represents relative discriminative resistance loss. This represents the artifact suppression loss.

[0189] The formula for pixel loss can be:

[0190] .

[0191] in, This represents a reference image.

[0192] The formula for perceived loss can be:

[0193] .

[0194] The formula for relative discriminant adversarial loss can be:

[0195] .

[0196] The formula for artifact suppression loss can be:

[0197] .

[0198] Where G represents the master generator, EMA represents the exponential moving average generator, and the weight update formula for the exponential moving average generator can be:

[0199] .

[0200] in, This represents the main generator parameters. This represents the smoothing parameter.

[0201] In this embodiment of the application, obtaining the relative discriminative adversarial loss between the reconstructed image and the reference image includes: extracting features from the reconstructed image to obtain a first semantic feature, and extracting features from the reference image to obtain a second semantic feature; inputting the first semantic feature and the second semantic feature into a discriminator, and using the discriminator to construct the relative discriminative adversarial loss.

[0202] In one optional embodiment, during the construction of the relative discriminative adversarial loss, the reconstruction system can first extract features from the reconstructed image to obtain a first semantic feature, and then extract features from the reference image to obtain a second semantic feature. The first and second semantic features are then input into the discriminator, and the discriminator is used to construct the relative discriminative adversarial loss.

[0203] In addition to addressing pixel loss, the reconstruction system adjusts its model by calculating the pixel-level differences between the predicted and reference images, ensuring that the reconstructed image is highly consistent with the original image numerically. Specifically, the reconstruction system compares each pixel value between the predicted and reference images, using the L1 or L2 norm to measure the difference between the two images. By minimizing pixel loss, the model learns how to accurately recover image details, ensuring that the pixel values ​​of the predicted image are as close as possible to the pixel values ​​of the reference image, thus guaranteeing a high numerical match between the reconstructed and target images.

[0204] To address perceptual loss, the reconstruction system can compare the perceptual differences between the predicted and reference images using features from intermediate layers of a pre-trained deep learning model. This ensures that the reconstructed image is not only numerically similar but also visually similar to the reference image. The reconstruction system feeds the predicted and reference images into the pre-trained deep model, extracts their respective feature representations, and then calculates the feature differences between corresponding layers. By minimizing this difference, the model learns how to generate an image that is visually close to the reference image, striving to make the reconstructed image as close as possible to the reference image in terms of structure, texture, and other aspects.

[0205] To address artifact suppression loss, the reconstruction system utilizes an exponential moving average (EMA) mechanism to monitor and suppress high-frequency artifacts that may occur during image super-resolution, such as checkerboard effects or ringing artifacts. The reconstruction system maintains an EMA model, whose parameters are updated using an exponential moving average of the master model's parameters. During training, the stability of the model's output can be evaluated by calculating the difference between images generated by the master model and those generated by the EMA model. This difference increases when the master model produces transient artifacts; therefore, by minimizing the artifact suppression loss, the model is incentivized to adjust its parameters in a way that reduces artifacts, resulting in more stable, artifact-free image output. This loss term helps ensure that the model, when generating super-resolution images, not only recovers details but also avoids visually unnatural artifacts, improving the overall image quality.

[0206] Figure 11 This is a schematic diagram illustrating an offline image reconstruction process according to an embodiment of this application, such as... Figure 11 As shown, the reconstruction system can trigger an image reconstruction task. After receiving the reconstruction task, the corresponding task processor can retrieve the original image to be reconstructed from the image library, call the image reconstruction model to reconstruct the original image, and then store the reconstructed target image in the image library.

[0207] Figure 12 This is a schematic diagram illustrating an online image reconstruction process according to an embodiment of this application, such as... Figure 12 As shown, users can upload the original image to be reconstructed. The corresponding task processor can first store the original image in the image library, and then call the API interface to use the image reconstruction model to reconstruct the original image to obtain the target image. The task processor can receive the target image returned by the image reconstruction model, update the image library with the target image, and return the target image to the user for easy viewing.

[0208] According to embodiments of this application, an image processing method is also provided. Figure 13 This is a flowchart illustrating an image processing method according to an embodiment of this application, such as... Figure 13As shown, the method may include the following steps:

[0209] Step S1302: When an input command is detected on the operation interface, the initial feature map of the original image is displayed on the operation interface.

[0210] The initial feature map is used to characterize the visual features of different pixels in the original image.

[0211] Step S1304: If a processing instruction is detected on the operation interface, the target image is displayed on the operation interface.

[0212] The target image is reconstructed from the original image using spatial feature maps. The spatial feature maps are used to characterize the semantic information and spatial location correlation between different pixels. The spatial feature maps are obtained by adjusting the enhanced feature maps using the attention weight matrix corresponding to the enhanced feature maps. The enhanced feature maps are used to characterize the dependencies between different pixels. The enhanced feature maps are constructed based on the attention weight matrix. The enhanced feature maps and the attention weight matrix are obtained by performing attention processing on features of different frequencies in the initial feature maps.

[0213] In one optional embodiment, to facilitate user viewing of the reconstruction results, upon detecting an input command applied to the user interface, the reconstruction system can first display an initial feature map of the original image on the user interface to characterize the visual features of different pixels in the original image. Upon detecting a processing command applied to the user interface, the target image is displayed on the user interface. The target image can be reconstructed from the original image based on a spatial feature map. The spatial feature map refers to the semantic information and spatial correlation between different pixels. The spatial feature map is obtained by adjusting the enhanced feature map using an attention weight matrix corresponding to the enhanced feature map. The enhanced feature map refers to the dependencies between different pixels. The enhanced feature map and the attention weight matrix are obtained by performing attention processing on features of different frequencies in the initial feature map.

[0214] According to embodiments of this application, an image processing apparatus for implementing the above-described image processing apparatus method is also provided. Figure 14 This is a structural block diagram of an image processing apparatus according to an embodiment of this application, such as... Figure 14 As shown, the device includes: a feature map acquisition module 1402, an attention processing module 1404, a feature map adjustment module 1406, and an image reconstruction module 1408.

[0215] The feature map acquisition module 1402 is used to acquire an initial feature map of the original image, wherein the initial feature map is used to represent the visual features of different pixels in the original image; the attention processing module 1404 is used to perform attention processing on features of different frequencies in the initial feature map to obtain an enhanced feature map of the original image and an attention weight matrix of the enhanced feature map, wherein the enhanced feature map is used to represent the dependency relationship between different pixels and is constructed based on the attention weight matrix; the feature map adjustment module 1406 is used to adjust the enhanced feature map based on the attention weight matrix to obtain a spatial feature map of the original image, wherein the spatial feature map is used to represent the semantic information and spatial location correlation between different pixels; and the image reconstruction module 1408 is used to reconstruct the original image based on the spatial feature map to obtain the target image.

[0216] The feature map acquisition module 1402, attention processing module 1404, feature map adjustment module 1406, and image reconstruction module 1408 described above correspond to steps S202 to S208 in the above embodiments. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. The modules or units described above may be hardware or software components stored in memory and processed by one or more processors. These modules may also be part of a device and may run in the server 10 provided in the above embodiments.

[0217] In this embodiment, the attention weight matrix includes a first weight matrix and a second weight matrix. The attention processing module 1404 is further configured to: input the initial feature map to the first decoupling module, perform attention processing on the initial feature map using the first decoupling module, and obtain the first feature and the first weight matrix of the original image; input the initial feature map to the second decoupling module, perform attention processing on the initial feature map using the second decoupling module, and obtain the second feature and the second weight matrix of the original image, wherein the frequency corresponding to the first feature in the spatial frequency domain is greater than the frequency corresponding to the second feature in the spatial frequency domain; and perform feature fusion on the first feature and the second feature to obtain an enhanced feature map.

[0218] In this embodiment, the attention processing module 1404 is further configured to: perform convolution processing on the initial feature map based on the first convolution parameters to obtain a first query matrix, a first key matrix, and a first value matrix of the initial feature map; obtain a first weight matrix based on the dot product of the first query matrix and the first key matrix; upsample the first weight matrix based on the resolution corresponding to the initial feature map to obtain an upsampled matrix; perform weighted processing on the upsampled matrix and the first value matrix to obtain a first weighted matrix; and perform convolution processing on the first weighted matrix to obtain a first feature.

[0219] In this embodiment, the attention processing module 1404 is further configured to: perform pooling processing on the initial feature map to obtain a pooled feature map; perform convolution processing on the initial feature map based on the second convolution parameters to obtain a second query matrix and a second key matrix of the initial feature map, and perform convolution processing on the initial feature map based on the third convolution parameters to obtain a second value matrix of the initial feature map; obtain a second weight matrix based on the dot product of the second query matrix and the second key matrix; perform weighted processing on the second weight matrix and the second value matrix to obtain a second weighted matrix; and perform convolution processing on the second weighted matrix to obtain a second feature.

[0220] In this embodiment of the application, the feature map adjustment module is further configured to: perform dimensional transformation on the enhanced feature map based on the first weight matrix to obtain an initial feature sequence, wherein the initial feature sequence is used to characterize the processing order of features corresponding to different pixels in the enhanced feature map; reconstruct the initial feature sequence based on the second weight matrix to obtain a target feature sequence; and reshape the target feature sequence based on the resolution of the enhanced feature map to obtain a spatial feature map.

[0221] In this embodiment, the feature map adjustment module is further configured to: upsample the first weight matrix based on the resolution corresponding to the initial feature map to obtain a third weight matrix; determine the first semantic saliency score and symmetric affinity matrix corresponding to any pixel based on the third weight matrix, wherein the first semantic saliency score is used to characterize the correlation between the pixel and other pixels, and the symmetric affinity matrix is ​​used to characterize the similarity between the pixel and other pixels; determine the pixel processing order of multiple pixels based on the first semantic saliency score and symmetric affinity matrix corresponding to multiple pixels in the enhanced feature map; and perform dimensionality transformation on the enhanced feature map based on the pixel processing order to construct an initial feature sequence.

[0222] In this embodiment, the feature map adjustment module is further configured to: cluster multiple pixels based on a symmetric affinity matrix to obtain multiple cluster sets; determine an importance index for a cluster set based on the first semantic saliency score of at least one pixel contained in any cluster set, wherein the importance index is used to characterize the importance of pixels in the cluster set in the image reconstruction process; sort the multiple cluster sets based on the importance index to obtain a processing order for the multiple cluster sets; and sort the multiple pixels based on the processing order and the first semantic saliency score of any pixel contained in any cluster set to obtain a pixel processing order.

[0223] In this embodiment of the application, the feature map adjustment module is further configured to: determine the second semantic saliency score of any pixel in the initial feature sequence based on the second weight matrix; select multiple target pixels from the multiple pixels contained in the initial feature sequence based on the second semantic saliency score to obtain a target feature sequence, wherein the second semantic saliency score corresponding to any target pixel is greater than a preset score threshold.

[0224] In this embodiment, the feature map adjustment module is further configured to: input the target feature sequence into a state space model, update the hidden state of the target feature sequence using the state space model to obtain an updated feature sequence; reshape the updated feature sequence based on a first operator to obtain a first feature map, and reshape the initial feature sequence based on a second operator to obtain a second feature map, wherein the resolution of the first feature map, the resolution of the second feature map, and the resolution of the enhanced feature map are the same; input the second feature map into a parallel convolution model, update the second feature map using the parallel convolution model to obtain a third feature map; and fuse the first feature map and the third feature map to obtain a spatial feature map.

[0225] In this embodiment, the apparatus further includes: a number acquisition module for acquiring the number of times the spatial feature map is constructed; a feature map evaluation module for evaluating the spatial feature map to obtain a feature map evaluation result, wherein the feature map evaluation result is used to characterize the accuracy of the spatial feature map; and a re-execution module for determining the spatial feature map as a new initial feature map and re-executing the frequency division and decoupling of the initial feature map to construct an enhanced feature map of the original image, adjusting the enhanced feature map based on the attention weight matrix to obtain the spatial feature map of the original image, until the feature map evaluation result corresponding to the spatial feature map characterizes the accuracy of the spatial feature map as greater than or equal to a preset accuracy threshold, or the number of constructions is greater than or equal to a preset number of constructions threshold.

[0226] In this embodiment of the application, the image reconstruction module is further configured to: input the spatial feature map to the upsampling module, perform upsampling processing on the spatial feature map using the upsampling module to obtain an upsampled feature map, wherein the resolution of the upsampled feature map is greater than the resolution of the spatial feature map; and perform image reconstruction on the original image based on the upsampled feature map to obtain the target image.

[0227] According to an embodiment of this application, a training apparatus for an image reconstruction model used to implement the above-described image reconstruction model training method is also provided. Figure 15 This is a structural block diagram of a training device for an image reconstruction model according to an embodiment of this application, such as... Figure 15As shown, the device includes: an image acquisition module 1502, an initial reconstruction module 1504, a function construction module 1506, and a model adjustment module 1508.

[0228] The image acquisition module 1502 is used to acquire training images and reference images; the initial reconstruction module 1504 is used to input the training images into the initial reconstruction model and use the initial reconstruction model to reconstruct the training images to obtain reconstructed images; the function construction module 1506 is used to construct the model loss function of the initial reconstruction model based on the reconstructed images and reference images; and the model adjustment module 1508 is used to adjust the initial reconstruction model using the model loss function to obtain an image reconstruction model, wherein the image reconstruction model is used in the above-mentioned method.

[0229] The image acquisition module 1502, initial reconstruction module 1504, function construction module 1506, and model adjustment module 1508 described above correspond to steps S902 to S908 in the above embodiments. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. The modules or units described above may be hardware or software components stored in memory and processed by one or more processors. These modules may also be part of a device and may run on the server 10 provided in the above embodiments.

[0230] In this embodiment of the application, the image acquisition module is further configured to: acquire a sample image; perform image degradation on the sample image to obtain a training image; perform upsampling on the sample image to obtain an upsampled image, and perform image enhancement on the sample image to obtain an enhanced image; and fuse the upsampled image and the enhanced image to obtain a reference image.

[0231] In this embodiment, the function construction module is further configured to: obtain pixel loss, perceptual loss, relative discriminative adversarial loss and artifact suppression loss between the reconstructed image and the reference image, wherein the relative discriminative adversarial loss is used to characterize the relative realism between the reconstructed image and the reference image, and the artifact suppression loss is used to characterize the degree of artifacts appearing in the reconstructed image; and weight the pixel loss, perceptual loss, relative discriminative adversarial loss and artifact suppression loss to obtain the model loss function.

[0232] In this embodiment of the application, the function construction module is further configured to: extract features from the reconstructed image to obtain a first semantic feature, and extract features from the reference image to obtain a second semantic feature; input the first semantic feature and the second semantic feature into the discriminator, and use the discriminator to construct a relative discriminative adversarial loss.

[0233] According to an embodiment of this application, another image processing apparatus for implementing the above-described image processing method is also provided. Figure 16This is a structural block diagram of another image processing apparatus according to an embodiment of this application, such as... Figure 16 As shown, the device includes: a first display module 1602 and a second display module 1604.

[0234] The first display module 1602 is used to display an initial feature map of the original image on the operation interface when an input command is detected on the operation interface. The initial feature map is used to characterize the visual features of different pixels in the original image. The second display module 1604 is used to display a target image on the operation interface when a processing command is detected on the operation interface. The target image is obtained by reconstructing the original image from the spatial feature map. The spatial feature map is used to characterize the semantic information and spatial location correlation between different pixels. The spatial feature map is obtained by adjusting the enhanced feature map with the attention weight matrix corresponding to the enhanced feature map. The enhanced feature map is used to characterize the dependency relationship between different pixels. The enhanced feature map is constructed based on the attention weight matrix. The enhanced feature map and the attention weight matrix are obtained by performing attention processing on features of different frequencies in the initial feature map.

[0235] The first display module 1602 and the second display module 1604 described above correspond to steps S1302 to S1304 in the above embodiments. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. The modules or units described above may be hardware or software components stored in memory and processed by one or more processors. The modules may also be part of a device and may run in the server 10 provided in the above embodiments.

[0236] The preferred embodiments involved in the above embodiments of this application are the same as the solutions, application scenarios and implementation processes provided in the above embodiments, and will not be repeated here.

[0237] Embodiments of this application may provide a computing device. Figure 17 This is a structural block diagram of a computing device according to an embodiment of this application. Figure 17 As shown, the computing device 1700 may include: one or more (one shown in the figure) processors 1702, memory 1704, memory controller, and peripheral interfaces.

[0238] The aforementioned computing device can be understood as an integrated intelligent terminal, including but not limited to servers, desktop computers, PCs (Personal Computers), and all-in-one model machines. Furthermore, the computing device may pre-install the model described in the above embodiments of this application.

[0239] Specifically, this computing device can pre-install various types of models, including but not limited to models in fields such as natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model choices. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API calling capabilities. Models can be called into created applications through API interfaces, and application management tools are provided to control and manage applications.

[0240] Furthermore, this computing device can also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI technologies), and basic control capabilities (providing enterprise-level basic control capabilities to ensure system security and efficient operation). Through these functions, it provides a comprehensive, integrated device for AI development, training, deployment, and application.

[0241] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0242] The processor can invoke an executable program stored in memory via a transmission device to execute any of the methods described in the above embodiments.

[0243] Embodiments of this application may provide an electronic device. Figure 18 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 18As shown, the electronic device may include: an input / output device 1802; a memory 1804; and a processor 1806, wherein the processor 1806 is connected to the input / output device 1802 and the memory 1804 via a bus 1808.

[0244] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the methods and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby implementing the methods in the above embodiments. The memory may include high-speed random access memory (RAM) and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memories. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks (LANs), mobile communication networks, and combinations thereof.

[0245] The processor can invoke an executable program stored in memory via a transmission device to execute any of the methods described in the above embodiments.

[0246] Those skilled in the art will understand that, Figure 18 The structure shown is illustrative. Electronic devices can also be smartphones (such as Android phones, iOS phones, etc.), tablets, PDAs, mobile internet devices (MIDs), PADs, and other terminal devices. This diagram does not limit the structure of the aforementioned electronic devices. For example, electronic devices may include more or fewer components (such as network interfaces, display devices, etc.) than shown in the diagram, or have a different configuration than shown in the diagram.

[0247] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: a flash drive, read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0248] Embodiments of this application also provide a computer-readable storage medium. Optionally, in this embodiment, the aforementioned computer-readable storage medium can be used to store program code executed by the method provided in the above embodiments.

[0249] Optionally, in this embodiment, the storage medium may be located in a computing device or an electronic device.

[0250] Optionally, in this embodiment, the computer-readable storage medium is configured to store an executable program. When the executable program runs, it controls the device where the computer-readable storage medium is located to perform any of the methods described in the above embodiments.

[0251] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product may include a computer program. When executed by a processor, the computer program implements the methods provided in the above embodiments.

[0252] Embodiments of this application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium. The non-volatile computer-readable storage medium can be used to store a computer program. When the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0253] Embodiments of this application also provide a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it implements the method provided in the above embodiments.

[0254] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0255] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are illustrative; for example, the division of units is a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined, integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling, direct coupling, or communication connection shown or discussed may be through some interfaces, indirect coupling of units or modules, or communication connection, and may be electrical or other forms.

[0256] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of this embodiment.

[0257] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0258] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0259] The above are preferred embodiments of this application. For those skilled in the art, various improvements and modifications can be made without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, include: An initial feature map of the original image is obtained, wherein the initial feature map is used to characterize the visual features of different pixels in the original image; Attention processing is applied to features of different frequencies in the initial feature map to obtain an enhanced feature map of the original image and an attention weight matrix of the enhanced feature map. The enhanced feature map is used to represent the dependency relationship between different pixels and is constructed based on the attention weight matrix. The enhanced feature map is adjusted based on the attention weight matrix to obtain the spatial feature map of the original image, wherein the spatial feature map is used to characterize the semantic information and spatial location correlation between different pixels; The target image is obtained by reconstructing the original image based on the spatial feature map.

2. The method according to claim 1, characterized in that, The attention weight matrix includes: a first weight matrix and a second weight matrix; attention processing is performed on features of different frequencies in the initial feature map to obtain the enhanced feature map of the original image, including: The initial feature map is input to the first decoupling module, and the first decoupling module is used to perform attention processing on the initial feature map to obtain the first feature and the first weight matrix of the original image. The initial feature map is input to the second decoupling module, and the initial feature map is subjected to attention processing by the second decoupling module to obtain the second feature and the second weight matrix of the original image, wherein the frequency of the first feature in the spatial frequency domain is greater than the frequency of the second feature in the spatial frequency domain. The first feature and the second feature are fused to obtain the enhanced feature map.

3. The method according to claim 2, characterized in that, The step of inputting the initial feature map into the first decoupling module and using the first decoupling module to perform attention processing on the initial feature map to obtain the first feature and the first weight matrix of the original image includes: The initial feature map is convolved based on the first convolution parameter to obtain the first query matrix, the first key matrix and the first value matrix of the initial feature map. The first weight matrix is ​​obtained based on the dot product of the first query matrix and the first key matrix; The first weight matrix is ​​upsampled based on the resolution corresponding to the initial feature map to obtain the upsampled matrix. The upsampling matrix and the first value matrix are weighted to obtain the first weighted matrix; The first weighted matrix is ​​convolved to obtain the first feature.

4. The method according to claim 2, characterized in that, The step of inputting the initial feature map into the second decoupling module and using the second decoupling module to perform attention processing on the initial feature map to obtain the second feature and the second weight matrix of the original image includes: The initial feature map is subjected to pooling processing to obtain a pooled feature map; The initial feature map is convolved based on the second convolution parameter to obtain the second query matrix and the second key matrix of the initial feature map, and then convolved based on the third convolution parameter to obtain the second value matrix of the initial feature map. The second weight matrix is ​​obtained based on the dot product of the second query matrix and the second key matrix; The second weight matrix and the second value matrix are weighted to obtain the second weighted matrix; The second weighted matrix is ​​convolved to obtain the second feature.

5. The method according to any one of claims 1-4, characterized in that, The step of adjusting the enhanced feature map based on the attention weight matrix to obtain the spatial feature map of the original image includes: The enhanced feature map is dimensionally transformed based on the first weight matrix to obtain an initial feature sequence, wherein the initial feature sequence is used to characterize the processing order of features corresponding to different pixels in the enhanced feature map; The initial feature sequence is reconstructed based on the second weight matrix to obtain the target feature sequence; The target feature sequence is reshaped based on the resolution of the enhanced feature map to obtain the spatial feature map.

6. The method according to claim 5, characterized in that, The step of performing dimensionality transformation on the enhanced feature map based on the first weight matrix to obtain an initial feature sequence includes: The first weight matrix is ​​upsampled based on the resolution corresponding to the initial feature map to obtain the third weight matrix; Based on the third weight matrix, a first semantic saliency score and a symmetric affinity matrix are determined for any pixel, wherein the first semantic saliency score is used to characterize the correlation between the pixel and other pixels, and the symmetric affinity matrix is ​​used to characterize the similarity between the pixel and other pixels. Based on the first semantic saliency score and symmetric affinity matrix corresponding to multiple pixels in the enhanced feature map, the pixel processing order of the multiple pixels is determined; The enhanced feature map is dimensionally transformed based on the pixel processing order to construct the initial feature sequence.

7. The method according to claim 6, characterized in that, The step of determining the pixel processing order of multiple pixels based on the first semantic saliency score and symmetric affinity matrix corresponding to multiple pixels in the enhanced feature map includes: Based on the symmetric affinity matrix, the multiple pixels are clustered to obtain multiple cluster sets; Based on the first semantic saliency score of at least one pixel in any cluster set, an importance index of the cluster set is determined, wherein the importance index is used to characterize the importance of the pixels in the cluster set in the image reconstruction process. The multiple cluster sets are sorted based on the importance index to obtain the processing order of the multiple cluster sets; Based on the processing order of the sets and the first semantic saliency score of any pixel contained in any cluster set, the multiple pixels are sorted to obtain the pixel processing order.

8. The method according to claim 5, characterized in that, The process of reconstructing the initial feature sequence based on the second weight matrix to obtain the target feature sequence includes: Based on the second weight matrix, determine the second semantic saliency score of any pixel in the initial feature sequence; Based on the second semantic saliency score, multiple target pixels are selected from the multiple pixels contained in the initial feature sequence to obtain a target feature sequence, wherein the second semantic saliency score corresponding to any target pixel is greater than a preset score threshold.

9. The method according to claim 5, characterized in that, The process of reshaping the target feature sequence to obtain the spatial feature map includes: The target feature sequence is input into a state space model, and the hidden state of the target feature sequence is updated using the state space model to obtain an updated feature sequence. The updated feature sequence is reshaped based on the first operator to obtain a first feature map, and the initial feature sequence is reshaped based on the second operator to obtain a second feature map, wherein the resolution of the first feature map, the resolution of the second feature map, and the resolution of the enhanced feature map are the same. The second feature map is input into a parallel convolution model, and the second feature map is updated using the parallel convolution model to obtain a third feature map; The first feature map and the third feature map are fused to obtain the spatial feature map.

10. The method according to any one of claims 1-4, characterized in that, After adjusting the enhanced feature map based on the attention weight matrix to obtain the spatial feature map of the original image, the method further includes: Obtain the number of times the spatial feature map is constructed; The spatial feature map is evaluated to obtain a feature map evaluation result, wherein the feature map evaluation result is used to characterize the accuracy of the spatial feature map; If the number of construction attempts is less than a preset threshold, or if the feature map evaluation result indicates that the accuracy of the spatial feature map is less than a preset accuracy threshold, the spatial feature map is determined as a new initial feature map, and frequency division and decoupling are re-executed on the initial feature map to construct an enhanced feature map of the original image. The enhanced feature map is then adjusted based on the attention weight matrix to obtain the spatial feature map of the original image. This process continues until the feature map evaluation result corresponding to the spatial feature map indicates that the accuracy of the spatial feature map is greater than or equal to the preset accuracy threshold, or the number of construction attempts is greater than or equal to the preset threshold.

11. The method according to any one of claims 1-4, characterized in that, The original image is reconstructed based on the spatial feature map to obtain the target image, including: The spatial feature map is input to the upsampling module, and the upsampling module is used to perform upsampling processing on the spatial feature map to obtain an upsampled feature map, wherein the resolution of the upsampled feature map is greater than the resolution of the spatial feature map; The target image is obtained by reconstructing the original image based on the upsampled feature map.

12. A training method for an image reconstruction model, characterized in that, include: Obtain training and reference images; The training image is input into the initial reconstruction model, and the initial reconstruction model is used to reconstruct the training image to obtain the reconstructed image; Based on the reconstructed image and the reference image, construct the model loss function of the initial reconstruction model; The initial reconstruction model is adjusted using the model loss function to obtain an image reconstruction model, wherein the image reconstruction model is used to perform the method described in any one of claims 1-11.

13. The method according to claim 12, characterized in that, Obtain training and reference images, including: Acquire sample images; The sample images are degraded to obtain the training images; The sample image is upsampled to obtain an upsampled image, and the sample image is then enhanced to obtain an enhanced image. The upsampled image and the enhanced image are fused to obtain the reference image.

14. The method according to claim 12, characterized in that, The model loss function for constructing the initial reconstruction model based on the reconstructed image and the reference image includes: The pixel loss, perceptual loss, relative discriminative adversarial loss, and artifact suppression loss between the reconstructed image and the reference image are obtained, wherein the relative discriminative adversarial loss is used to characterize the relative realism between the reconstructed image and the reference image, and the artifact suppression loss is used to characterize the degree of artifacts appearing in the reconstructed image; The model loss function is obtained by weighting the pixel loss, the perceptual loss, the relative discriminative adversarial loss, and the artifact suppression loss.

15. The method according to claim 14, characterized in that, Obtaining the relative discriminative adversarial loss between the reconstructed image and the reference image includes: Feature extraction is performed on the reconstructed image to obtain a first semantic feature, and feature extraction is performed on the reference image to obtain a second semantic feature; The first semantic feature and the second semantic feature are input into the discriminator, and the discriminator is used to construct the relative discriminative adversarial loss.

16. An image processing method, characterized in that, include: Upon detecting an input command applied to the operation interface, an initial feature map of the original image is displayed on the operation interface, wherein the initial feature map is used to characterize the visual features of different pixels in the original image. Upon detecting a processing instruction applied to the operation interface, a target image is displayed on the operation interface. The target image is reconstructed from the original image using a spatial feature map. The spatial feature map is used to characterize the semantic information and spatial location correlation between different pixels. The spatial feature map is obtained by adjusting the enhanced feature map using an attention weight matrix corresponding to the enhanced feature map. The enhanced feature map is used to characterize the dependency relationship between different pixels. The enhanced feature map is constructed based on the attention weight matrix. The enhanced feature map and the attention weight matrix are obtained by performing attention processing on features of different frequencies in the initial feature map.

17. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor, connected to a memory via a bus, is used to run the program, wherein the program, when running, performs the method according to any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored executable program, wherein, when the executable program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 15.

19. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 15.

Images