Image processing method, device, equipment, medium and computer program product

Abstract

The present application relates to network media technology, and more particularly to an image processing method, device, equipment, medium and computer program product. The method comprises: extracting features from a first image to obtain initial features of the first image; performing resolution enhancement processing on the initial features, and reconstructing a second image based on the processed features; obtaining key area marking information of the second image under a plurality of key information representation methods; for each key information representation method, extracting features of the key area marking information under the key information representation method through a priori feature extraction network corresponding to the key information representation method and obtained by network structure search processing, and obtaining key area prior features corresponding to the key information representation method; fusing the processed features with the key area prior features corresponding to the plurality of key information representation methods to obtain fused features; and reconstructing the image based on the fused features to obtain a target image. This method can improve image quality.

Description

Technical Field

[0001] This application relates to the field of network media technology, and in particular to an image processing method, apparatus, device, medium and computer program product. Background Technology

[0002] With the development of computer technology, artificial intelligence (AI) has emerged. AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, enabling them to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0003] Artificial intelligence (AI) technology is widely used in image processing, for example, to restore image quality. Traditional techniques typically employ manually designed neural networks for image quality restoration. However, manually designed neural networks often have poor image restoration performance, resulting in low-quality restored images. Summary of the Invention

[0004] Therefore, it is necessary to provide an image processing method, apparatus, device, medium, and computer program product that can improve the quality of the recovered image in response to the above-mentioned technical problems.

[0005] On one hand, this application provides an image processing method, the method comprising:

[0006] Feature extraction is performed on the first image to be restored to obtain the initial features of the first image;

[0007] The initial features are subjected to resolution enhancement processing, and a second image with the image content of the first image and a higher resolution than the first image is reconstructed based on the processed features.

[0008] Obtain key region marking information of the second image under multiple key information representation methods;

[0009] For each key information representation method, the features of the key region labeling information under the key information representation method are extracted by the prior feature extraction network corresponding to the key information representation method and obtained by network structure search processing, so as to obtain the key region prior features corresponding to the key information representation method.

[0010] The processed features and the prior features of key regions corresponding to the various key information representation methods are fused to obtain fused features;

[0011] Based on the fusion features, the image is reconstructed to obtain the target image restored from the first image.

[0012] On the other hand, this application provides an image processing apparatus, the apparatus comprising:

[0013] The extraction module is used to extract features from the first image to be restored, and obtain the initial features of the first image;

[0014] The reconstruction module is used to perform resolution enhancement processing on the initial features and reconstruct a second image based on the processed features, which has the image content of the first image and has a higher resolution than the first image.

[0015] The acquisition module is used to acquire key region marking information of the second image under multiple key information representation methods;

[0016] The extraction module is also used to extract the features of the key region labeling information under the key information representation method by using the prior feature extraction network corresponding to the key information representation method and obtained by network structure search processing, for each key information representation method, and to obtain the key region prior features corresponding to the key information representation method.

[0017] The fusion module is used to fuse the processed features and the prior features of key regions corresponding to the various key information representation methods to obtain fused features;

[0018] The reconstruction module is also used to reconstruct the image based on the fusion features to obtain the target image restored from the first image.

[0019] In one embodiment, the reconstruction module is further configured to perform upsampling processing on the initial features through an upsampling network obtained by network structure search and matching the first image, to obtain upsampling features; and reconstruct a second image with the image content of the first image and a resolution higher than that of the first image based on the upsampling features.

[0020] In one embodiment, the fusion module is further configured to fuse the upsampled features and the prior features of key regions corresponding to the various key information representation methods through a feature fusion network obtained by network structure search processing and matched with the first image, to obtain fused features.

[0021] In one embodiment, the key region marking information under multiple key information representation methods includes at least two of the following: a key region segmentation map under a first representation method, a key region heat map under a second representation method, and key component dictionary information under a third representation method.

[0022] In one embodiment, the prior feature extraction network corresponding to each key information representation method belongs to a sub-network of the trained image quality restoration model; the trained image quality restoration model further includes an upsampling network and a feature fusion network; the resolution enhancement processing is performed by the upsampling network; the fused features are obtained by the feature fusion network; the apparatus further includes:

[0023] A training module is used to acquire multiple sample images; the multiple sample images have similar image quality to the first image; based on the multiple sample images, a network structure search is performed in a preset network structure search space, and an initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network are constructed based on the searched network structure; the network parameters of the initial upsampling network, each of the initial prior feature extraction networks, and the initial feature fusion network are updated using the multiple sample images to obtain the upsampling network, the prior feature extraction network corresponding to each key information representation method, and the feature fusion network.

[0024] In one embodiment, the preset network structure search space includes multiple network layers connected in sequence; each network layer includes multiple cells; the training module is further configured to, in the current iteration, determine the current network layer from the multiple network layers based on the multiple sample images, and for each cell in the current network layer, determine the output features of associated cells that have path constraint relationships in at least some prior network layers, and fuse the output features of each associated cell based on the path weight factors between the cell and each associated cell to obtain the input features of the cell; perform corresponding feature processing on the input features of the cell to obtain the output features of the cell, and pass the output features of the cell to the next layer of the network. The process involves taking the next network layer as the current network layer, returning to the step of determining the output features of associated cells with path constraints in at least some prior network layers for each cell in the current network layer, and continuing until each network layer has been traversed in this round to obtain the output result of this round; adjusting each path weight factor based on the output result of this round, and taking the next round as this round to continue iterating until the search stopping condition is met to obtain the searched network structure, and constructing an initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network based on the searched network structure.

[0025] In one embodiment, each cell includes a plurality of sequentially connected feature processing blocks; the training module is further configured to determine the operation weight factors of the candidate operations corresponding to each feature processing block in the cell; determine the input features of each feature processing block based on the input features of the cell, and execute each candidate operation on the input features of the feature processing blocks respectively to obtain the output features of the feature processing blocks corresponding to each candidate operation; fuse the output features of each feature processing block corresponding to each candidate operation according to the operation weight factors of each candidate operation to obtain the output features of the feature processing block; fuse the output features of each feature processing block in the cell to obtain the output features of the cell; adjust each path weight factor and each operation weight factor based on the output results of the current round, and take the next round as the current round to continue the iteration.

[0026] In one embodiment, the training module is further configured to, for each feature processing block in the cell, take the input features of the cell and the output features of the preceding feature processing block in the cell as the input features of the feature processing block; the preceding feature processing block is the feature processing block that precedes the current feature processing block among the various feature processing blocks of the cell.

[0027] In one embodiment, the target image is obtained by outputting a trained image quality restoration model; the training module is further used to acquire sample images; the sample images include a first sample image and a second sample image with the same content as the first sample image; the image quality of the second sample image is higher than that of the first sample image; the first sample image includes a first single-factor sample image and a first multi-factor sample image; the first single-factor sample image is a first sample image generated based on multiple different single factors; the first multi-factor sample image is a first sample image generated based on the multiple different single factors; the image quality restoration model to be trained is trained using the sample images to obtain a trained image quality restoration model; the trained image quality restoration model includes a prior feature extraction network corresponding to each key information representation method.

[0028] In one embodiment, the training module is further configured to input the first sample image into the image quality restoration model to be trained, so as to restore the quality of the first sample image through the image quality restoration model to be trained, and obtain the restored image; determine the target loss value based on the difference between the restored image and the second sample image; and iteratively train the image quality restoration model to be trained in the direction of reducing the target loss value until the iteration stopping condition is reached, so as to obtain the trained image quality restoration model.

[0029] In one embodiment, the training module is further configured to determine a first loss value based on the pixel difference between the recovered image and the second sample image; extract a first feature of the recovered image and a second feature of the second sample image respectively; determine a second loss value based on the feature difference between the first feature and the second feature; and determine a target loss value based on the first loss value and the second loss value.

[0030] In one embodiment, the target image is obtained by outputting a trained image quality restoration model; the image quality restoration model includes an initial feature extraction network; the initial feature extraction network includes convolutional layers and residual layers; the residual layer includes at least one residual fully connected unit; the extraction module is further configured to input the first image to be restored into the convolutional layer, so as to perform convolution on the first image to be restored through the convolutional layer to obtain convolutional features; and input the convolutional features into the residual layer, so as to process the convolutional features through the at least one residual fully connected unit in the residual layer to obtain the initial features of the first image.

[0031] On the other hand, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0032] Feature extraction is performed on the first image to be restored to obtain the initial features of the first image;

[0033] The initial features are subjected to resolution enhancement processing, and a second image with the image content of the first image and a higher resolution than the first image is reconstructed based on the processed features.

[0034] Obtain key region marking information of the second image under multiple key information representation methods;

[0035] For each key information representation method, the features of the key region labeling information under the key information representation method are extracted by the prior feature extraction network corresponding to the key information representation method and obtained by network structure search processing, so as to obtain the key region prior features corresponding to the key information representation method.

[0036] The processed features and the prior features of key regions corresponding to the various key information representation methods are fused to obtain fused features;

[0037] Based on the fusion features, the image is reconstructed to obtain the target image restored from the first image.

[0038] On the other hand, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0039] Feature extraction is performed on the first image to be restored to obtain the initial features of the first image;

[0040] The initial features are subjected to resolution enhancement processing, and a second image with the image content of the first image and a higher resolution than the first image is reconstructed based on the processed features.

[0041] Obtain key region marking information of the second image under multiple key information representation methods;

[0042] For each key information representation method, the features of the key region labeling information under the key information representation method are extracted by the prior feature extraction network corresponding to the key information representation method and obtained by network structure search processing, so as to obtain the key region prior features corresponding to the key information representation method.

[0043] The processed features and the prior features of key regions corresponding to the various key information representation methods are fused to obtain fused features;

[0044] Based on the fusion features, the image is reconstructed to obtain the target image restored from the first image.

[0045] On the other hand, this application provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0046] Feature extraction is performed on the first image to be restored to obtain the initial features of the first image;

[0047] The initial features are subjected to resolution enhancement processing, and a second image with the image content of the first image and a higher resolution than the first image is reconstructed based on the processed features.

[0048] Obtain key region marking information of the second image under multiple key information representation methods;

[0049] For each key information representation method, the features of the key region labeling information under the key information representation method are extracted by the prior feature extraction network corresponding to the key information representation method and obtained by network structure search processing, so as to obtain the key region prior features corresponding to the key information representation method.

[0050] The processed features and the prior features of key regions corresponding to the various key information representation methods are fused to obtain fused features;

[0051] Based on the fusion features, the image is reconstructed to obtain the target image restored from the first image.

[0052] The aforementioned image processing methods, apparatus, devices, media, and computer program products, by extracting features from a first image to be restored, can obtain initial features of the first image. By performing resolution enhancement processing on the initial features, processed features with higher resolution than the initial features can be obtained. Based on the processed features, a second image containing the image content of the first image and with higher resolution can be reconstructed. Key region marking information of the second image under various key information representation methods is obtained. For each key information representation method, features of the key region marking information under that key information representation method can be extracted using a prior feature extraction network corresponding to that key information representation method and obtained through network structure search processing. This yields the prior features of the key regions corresponding to the key information representation method. Furthermore, the processed features and the prior features of the key regions corresponding to the various key information representation methods are fused to obtain fused features. Based on the fused features, image reconstruction can be performed to obtain a target image restored from the first image, with image quality higher than both the first and second images. Compared to traditional manually designed networks, this application uses a prior feature extraction network obtained through network structure search processing corresponding to the key information representation method. This network can extract key region labeling information with better performance under the key information representation method. At the same time, by fusing the features after resolution enhancement and the key region prior features corresponding to various key information representation methods, richer fused features can be obtained. Based on these fused features, a target image with higher image quality can be reconstructed, thereby improving the quality of the recovered image. Attached Figure Description

[0053] Figure 1 This is an application environment diagram of an image processing method in one embodiment;

[0054] Figure 2 This is a flowchart illustrating an image processing method in one embodiment;

[0055] Figure 3 This is a schematic diagram illustrating the principle of determining each cell in the network structure in one embodiment;

[0056] Figure 4 This is a schematic diagram illustrating the principle of obtaining the output features of each cell in one embodiment;

[0057] Figure 5 This is a schematic diagram of the model structure of an image quality restoration model in one embodiment;

[0058] Figure 6 This is a flowchart illustrating the image processing method in another embodiment;

[0059] Figure 7 This is a structural block diagram of an image processing device in one embodiment;

[0060] Figure 8 This is a structural block diagram of an image processing apparatus in another embodiment;

[0061] Figure 9 This is an internal structural diagram of a computer device in one embodiment;

[0062] Figure 10 This is a diagram of the internal structure of a computer device in another embodiment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] The image processing method provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on other servers. Terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, smart TVs, portable wearable devices, and vehicle terminals. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. Terminal 102 and server 104 can be directly or indirectly connected via wired or wireless communication; this application does not impose any restrictions on this connection.

[0065] Server 104 can extract features from the first image to be restored, obtain initial features of the first image, perform resolution enhancement processing on the initial features, and reconstruct a second image with the image content of the first image and a higher resolution based on the processed features. Server 104 can obtain key region marking information of the second image under various key information representation methods. For each key information representation method, it extracts features of the key region marking information under the key information representation method through a prior feature extraction network corresponding to the key information representation method and obtained by network structure search processing, thus obtaining the key region prior features corresponding to the key information representation method. Server 104 can fuse the processed features and the key region prior features corresponding to various key information representation methods to obtain fused features, and reconstruct the image based on the fused features to obtain the target image restored from the first image.

[0066] It is understood that terminal 102 can provide a first image to be recovered, and server 104 can perform feature extraction on the first image to be recovered. Server 104 can also directly obtain the first image to be recovered from local storage. This embodiment does not limit this; it is understood that... Figure 1 The application scenarios shown are for illustrative purposes only and are not limited to these.

[0067] In one embodiment, such as Figure 2 As shown, an image processing method is provided. This method can be applied to a computer device, which can be a terminal or a server, and is executed independently by the terminal or server. It is understood that the method of this application can also be implemented through interaction between a terminal and a server. This embodiment uses the application of this method to a computer device as an example for illustration, and includes the following steps:

[0068] Step 202: Extract features from the first image to be restored to obtain the initial features of the first image.

[0069] The first image is an image of low quality, which can be due to at least one of the following: image blurring, image containing noise, low image resolution, or image compression. It is understood that the image processing method of this application can restore the image quality of the first image with low quality. The initial features are image features directly extracted from the first image without undergoing feature processing.

[0070] In one embodiment, the image may include images of various scenes, such as at least one of facial images, human images, animal images, plant images, landscape images, and building images.

[0071] Specifically, the computer device can acquire a first image of low quality to be restored, and perform feature extraction processing on the first image to be restored in order to obtain the initial features of the first image from the first image to be restored.

[0072] In one embodiment, the computer device performs feature extraction processing on the first image to be restored. Specifically, it may first perform convolution processing on the first image to extract convolutional features from the first image. Then, the computer device may input the convolutional features extracted from the first image into a residual network to perform further feature processing on the convolutional features extracted from the first image through the residual network, thereby obtaining the initial features of the first image.

[0073] Step 204: Perform resolution enhancement processing on the initial features, and reconstruct a second image based on the processed features, which has the image content of the first image and a higher resolution than the first image.

[0074] Resolution upscaling refers to the process of increasing the resolution of the initial features. The processed features are the image features after resolution upscaling the initial features. The resolution of the processed features is higher than the resolution of the initial features before resolution upscaling. It can be understood that feature resolution refers to the resolution of the feature map. Image content refers to the content described by the scene in the image. The second image is an image directly reconstructed based on the resolution upscaled features. While the resolution of the second image is higher than that of the first image, meaning the image quality of the second image is higher than that of the first image, the second image only exists during the quality restoration process of the first image and is not used as the final output image of the quality restoration. Reconstruction refers to image reconstruction, that is, the process of decoding the encoded image features to obtain a perceptible image.

[0075] Specifically, the initial features of the first image have a low resolution. To obtain higher resolution image features, the computer device can perform resolution upscaling on the initial features to obtain upscaled features. Then, the computer device can perform image reconstruction based on the upscaled features, which have a higher resolution than the initial features, to reconstruct a second image that contains the image content of the first image and has a higher resolution than the first image.

[0076] In one embodiment, the computer device may input the initial features of a first image into a resolution upscaling network to perform resolution upscaling on the initial features of the first image, thereby obtaining resolution-upgraded features. Then, the computer device may perform image reconstruction based on the resolution-upgraded features to obtain a second image that contains the image content of the first image but has a higher resolution than the first image.

[0077] In one embodiment, the resolution-enhancing network can be either a network obtained through network structure search or a network pre-constructed manually. It is understood that a manually pre-constructed network has a pre-defined network structure, while a network obtained through network structure search is constructed based on a network structure searched from a pre-defined network structure search space.

[0078] Step 206: Obtain key region marking information of the second image under multiple key information representation methods.

[0079] Key information representation methods are used to represent key information in an image. Key information refers to information about key regions in an image. Key regions are relatively important areas in an image. Key region labeling information is used to label key regions in an image. For example, if the image is a face image, the key regions of the face image can include at least one of the regions where the eyes, nose, and mouth are located. The key region labeling information of this face image can be information used to label at least one of the regions where the eyes, nose, and mouth are located. It can be understood that key regions of an image can be represented by various different key information representation methods. For example, for a face image, the region where the mouth is located can be represented by various different key information representation methods. It can be understood that by labeling key regions of an image using key region labeling information, the labeled key regions can be given a higher recovery weight than other non-key regions in the subsequent image quality restoration process, so as to give more attention to the key regions of the image during the image quality restoration process, thereby achieving the focus on restoring the key regions of the image.

[0080] Specifically, the computer equipment can acquire key region marking information of the second image under various key information representation methods.

[0081] In one embodiment, the key region marking information under multiple key information representation methods may specifically include at least two of the following: a key region segmentation map under a first representation method, a key region heat map under a second representation method, and key component dictionary information under a third representation method.

[0082] In one embodiment, if the image is a face image, the key region segmentation map may include a face key region segmentation map, the key region heatmap may specifically include a face key region heatmap, and the key component dictionary information may include face key component dictionary information.

[0083] In the above embodiments, key regions of an image are marked by at least two types of key region marking information, including key region segmentation map, key region heat map, and key component dictionary information. This allows for more accurate location of key regions in the image that need to be restored, thereby improving the image quality obtained from subsequent image restoration.

[0084] Step 208: For each key information representation method, extract the features of key region labeling information under the key information representation method through the prior feature extraction network corresponding to the key information representation method and obtained by network structure search processing, and obtain the key region prior features corresponding to the key information representation method.

[0085] The network structure search process refers to the method of searching for a network structure in a predefined network structure search space and constructing a corresponding neural network based on the searched network structure. It can be understood that the predefined network structure search space may include multiple paths for constructing the network. Through the network structure search process of this application, an optimal path can be searched from these multiple paths, and the network structure can be determined based on the searched path. The prior feature extraction network is used to extract prior features from key region labeling information. Prior features are features that satisfy constraints constructed based on prior knowledge. Key region prior features are prior features extracted from key region labeling information.

[0086] Specifically, the computer device deploys prior feature extraction networks corresponding to various key information representation methods, each obtained through network structure search processing. For each key information representation method, the computer device can input the key region labeling information under that key information representation method into the prior feature extraction network corresponding to that key information representation method, obtained through network structure search processing. The prior feature extraction network then performs feature extraction processing on the key region labeling information under that key information representation method, obtaining the features of the key region labeling information under that key information representation method. The computer device can use these features of the key region labeling information under that key information representation method as the prior features of the key region corresponding to that key information representation method.

[0087] Step 210: The processed features and the prior features of key regions corresponding to various key information representation methods are fused to obtain fused features.

[0088] Among them, the fusion feature is the feature obtained by fusing the processed feature with the prior features of the key regions corresponding to various key information representation methods. It can be understood that the fusion feature includes both the information of the processed feature and the information of the prior features of the key regions corresponding to various key information representation methods.

[0089] Specifically, the computer device can perform feature fusion processing on the processed features and the prior features of key regions corresponding to various key information representation methods, and use the features obtained after feature fusion processing as fused features.

[0090] In one embodiment, the computer device fuses the processed features and the prior features of key regions corresponding to various key information representation methods. Specifically, this can be achieved by concatenating the processed features and the prior features of key regions corresponding to various key information representation methods, or by performing a weighted operation on the processed features and the prior features of key regions corresponding to various key information representation methods. This application does not specifically limit the method of feature fusion.

[0091] Step 212: Reconstruct the image based on the fusion features to obtain the target image restored from the first image.

[0092] The target image is the image that serves as the objective; it can be understood as the final output image of the image quality restoration process.

[0093] Specifically, the computer device can reconstruct the image based on fusion features and use the reconstructed image as the target image recovered from the first image. It can be understood that the image content of the target image after image quality recovery is the same as the image content of the first and second images, and the image quality of the target image after image quality recovery is higher than that of the first and second images.

[0094] In the above image processing method, feature extraction is performed on the first image to be restored to obtain initial features of the first image. Resolution enhancement processing is then applied to these initial features to obtain processed features with higher resolution than the initial features. Based on these processed features, a second image containing the image content of the first image and with higher resolution can be reconstructed. Key region markers of the second image under various key information representation methods are obtained. For each key information representation method, a prior feature extraction network corresponding to that method, obtained through network structure search processing, can extract features of the key region markers under that key information representation method, obtaining the key region prior features corresponding to that key information representation method. Then, the processed features and the key region prior features corresponding to the various key information representation methods are fused to obtain fused features. Based on these fused features, image reconstruction is performed to obtain a target image restored from the first image, with image quality higher than both the first and second images. Compared to traditional manually designed networks, this application uses a prior feature extraction network obtained through network structure search processing corresponding to the key information representation method. This network can extract key region labeling information with better performance under the key information representation method. At the same time, by fusing the features after resolution enhancement and the key region prior features corresponding to various key information representation methods, richer fused features can be obtained. Based on these fused features, a target image with higher image quality can be reconstructed, thereby improving the quality of the recovered image.

[0095] In one embodiment, the process of performing resolution upsampling on initial features and reconstructing a second image with image content of a first image and a higher resolution than the first image based on the processed features includes: performing upsampling on the initial features through an upsampling network obtained by network structure search and matching the first image to obtain upsampled features; and reconstructing a second image with image content of the first image and a higher resolution than the first image based on the upsampled features.

[0096] Upsampling networks are one type of resolution enhancement network mentioned above. It's understood that resolution enhancement networks can also include other networks that can improve feature resolution, such as upconvolutional networks. Upsampled features are features obtained by upsampling the initial features; therefore, the resolution of upsampled features is higher than that of the initial features.

[0097] Specifically, the computer device can input initial features into an upsampling network that is obtained through network structure search processing and matches the first image. The upsampling network, obtained through network structure search processing and matching the first image, upsamples the initial features and outputs upsampled features. Then, based on the upsampled features, the computer device can reconstruct a second image that has the image content of the first image and has a higher resolution than the first image.

[0098] In one embodiment, a computer device can upsample initial features using an upsampling network obtained through network structure search and matched with the first image, to obtain upsampled features. Based on these upsampled features, a second image with the image content of the first image and a higher resolution than the first image can be reconstructed. Furthermore, the computer device can acquire key region marking information of the second image under various key information representation methods. For each key information representation method, features of the key region marking information under that key information representation method are extracted using a prior feature extraction network obtained through network structure search, corresponding to that key information representation method, to obtain the key region prior features corresponding to that key information representation method. The computer device can input the upsampled features and the key region prior features corresponding to the various key information representation methods into a manually constructed feature fusion network. The network fuses the upsampled features and the key region prior features corresponding to the various key information representation methods to obtain fused features. Based on these fused features, image reconstruction is performed to obtain the target image restored from the first image.

[0099] In the above embodiments, the initial features are upsampled by an upsampling network obtained by network structure search and matching the first image. Compared with the traditional feature processing network constructed manually, richer upsampled features can be obtained. Based on the richer upsampled features, a second image with higher image quality can be reconstructed.

[0100] In one embodiment, the processed features and the prior features of key regions corresponding to various key information representation methods are fused to obtain fused features. This includes fusing the upsampled features and the prior features of key regions corresponding to various key information representation methods through a feature fusion network obtained by network structure search and matching the first image to obtain fused features.

[0101] Specifically, the computer device can input the upsampled features and the prior features of key regions corresponding to various key information representation methods into a feature fusion network that is obtained by network structure search processing and matches the first image. The feature fusion network that is obtained by network structure search processing and matches the first image can fuse the upsampled features and the prior features of key regions corresponding to various key information representation methods to obtain fused features.

[0102] In the above embodiments, the feature fusion network obtained by network structure search and matching the first image can better fuse the upsampled features and the prior features of key regions corresponding to various key information representation methods, compared with the traditional feature fusion network constructed manually, to obtain more accurate fused features, thereby further improving the quality of the recovered image.

[0103] In one embodiment, the prior feature extraction network corresponding to each key information representation method belongs to a sub-network of the trained image quality restoration model; the trained image quality restoration model further includes an upsampling network and a feature fusion network; the resolution enhancement processing is performed by the upsampling network; the fusion features are obtained by the feature fusion network; the steps to obtain the trained image quality restoration model include: acquiring multiple sample images; the multiple sample images have similar image quality to the first image; based on the multiple sample images, performing network structure search processing in a preset network structure search space, and constructing an initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network based on the searched network structure; updating the network parameters of the initial upsampling network, each initial prior feature extraction network, and the initial feature fusion network through multiple sample images to obtain the upsampling network, the prior feature extraction network corresponding to each key information representation method, and the feature fusion network.

[0104] Here, sample images are the images used to train the image quality restoration model during the training process. In other words, sample images are the training data used to train the image quality restoration model. The statement that multiple sample images have similar image quality to the first image means that the image quality of multiple sample images is similar to that of the first image. The initial upsampling network is an upsampling network built based on the searched network structure, and its network parameters still need to be updated. The initial prior feature extraction network is a prior feature extraction network built based on the searched network structure, and its network parameters still need to be updated. The initial feature fusion network is a feature fusion network built based on the searched network structure, and its network parameters still need to be updated.

[0105] Specifically, the computer device can acquire multiple sample images with image quality similar to the first image, and based on these sample images, perform a network structure search within a preset network structure search space to obtain the searched network structure. Then, the computer device can construct an initial upsampling network, initial prior feature extraction networks corresponding to each key information representation method, and an initial feature fusion network based on the searched network structure. The computer device can iteratively update the network parameters of the initial upsampling network, each initial prior feature extraction network, and the initial feature fusion network using multiple sample images until the iteration stops, thus obtaining the upsampling network, the prior feature extraction networks corresponding to each key information representation method, and the feature fusion network.

[0106] It can be understood that the process of searching for and processing the network structure in the preset network structure search space to obtain the network structure, as well as the process of updating the network parameters of the initial upsampling network, each initial prior feature extraction network, and the initial feature fusion network obtained based on the network structure, all belong to the training process of the image quality restoration model.

[0107] In the above embodiments, by using multiple sample images, a network structure search is performed in a preset network structure search space to find the optimal network structure most suitable for image quality restoration of the sample images. Based on the searched network structure, an optimal initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network can be constructed. Furthermore, by updating the network parameters of the initial upsampling network, each initial prior feature extraction network, and the initial feature fusion network using multiple sample images, an upsampling network with better image quality restoration effect, an prior feature extraction network corresponding to each key information representation method, and a feature fusion network can be obtained, further improving the quality of the restored image.

[0108] In one embodiment, a preset network structure search space includes multiple network layers connected sequentially; each network layer includes multiple cells; based on multiple sample images, a network structure search process is performed in the preset network structure search space, and an initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network are constructed based on the searched network structure, including: in this iteration, based on multiple sample images, the current network layer is determined from multiple network layers; for each cell in the current network layer, the output features of associated cells that have path constraint relationships in at least some prior network layers are determined; and based on the path weight factors between the cell and each associated cell, the output features of each associated cell are fused to obtain the input features of the cell; for the cell... The input features are processed accordingly to obtain the output features of the cells. These output features are then input into the next layer of the network to the cells that have path constraints with the current cell. The next layer is used as the current layer. The process of determining the output features of the cells that have path constraints with the current cell in at least some of the previous layers is repeated until each layer has been traversed in this round, yielding the output result. The path weights are adjusted based on the output result, and the next round is used as the current round. The iteration continues until the search stopping condition is met, resulting in the searched network structure. Based on the searched network structure, an initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network are constructed.

[0109] The number of network layers can be understood as representing the network depth of an image quality restoration model, with cells being the basic building blocks of the network structure. A preceding network layer refers to a network layer within the predefined network structure search space that precedes the current network layer. Path constraints characterize the constraint relationships between feature transfer paths between cells in adjacent network layers. For example, if the output feature of cell A in the fourth network layer can only be transferred to cells A1, A2, and A3 in the fifth layer via three feature transfer paths, but there is no feature transfer path to transfer the output feature of cell A to cells B1, B2, and B3, then cell A in the fourth network layer has a path constraint relationship with cells A1, A2, and A3 in the fifth layer, but not with cells B1, B2, and B3 in the fifth layer. Associated cells are associated cells in adjacent network layers that have path constraint relationships. Path weight factors are parameters used to characterize the probability of inputting image features from cells in the previous network layer to associated cells in the next network layer.

[0110] In one embodiment, the search stopping condition can be that the number of iterative searches reaches a preset number of searches, or the path weight factor stops updating.

[0111] Specifically, in this iteration, the computer device can determine the current network layer from multiple network layers based on multiple sample images. For each cell in the current network layer, it determines the output features of associated cells that have path constraints with the cell in at least some of the preceding network layers. Based on the path weight factors between the cell and its associated cells obtained from the previous iteration, it fuses the output features of each associated cell to obtain the cell's input features. The computer device can perform corresponding feature processing on the cell's input features to obtain the cell's output features, and input the cell's output features into the associated cells with path constraints in the next layer of the network. The computer device can use the next layer as the current network layer and return to the step of determining the output features of associated cells that have path constraints with the cell in at least some of the preceding network layers for each cell in the current network layer to continue iterating until each network layer has been traversed in this round, obtaining the output result of this round. Furthermore, the computer device can adjust the weight factors of each path based on the output results of the current round, and take the next round as the current round. In the current iteration, based on multiple sample images, the current network layer is determined from multiple network layers. For each cell in the current network layer, the output features of the associated cells that have path constraint relationships in at least some of the preceding network layers are determined iteratively until the search stopping condition is met, and the searched network structure is obtained. Based on the searched network structure, an initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network are constructed.

[0112] It is understandable that when the search stopping condition is met, the computer device can select a target cell from each network layer based on the path weight factor, and determine the searched network structure based on the selected target cell. For example, the output features of the target cell selected from the first network layer can be used as the input features of the target cell selected from the second network layer, the output features of the target cell selected from the second network layer can be used as the input features of the target cell selected from the third network layer, and so on, until it is passed to multiple network layers in the preset network structure search space. The computer device can use the output features of the target cell selected from the last network layer as the output of the corresponding network generated based on that network structure.

[0113] In one embodiment, each of the multiple network layers may include multiple optional upsampling factors. For example, the network may have six layers, and each layer may include four optional upsampling factors, i.e., {1, 2, 4, 8}. The upsampling factor is a factor that increases the resolution of the features. For instance, an upsampling factor of 2 means the feature resolution is increased by a factor of 2, and an upsampling factor of 8 means the feature resolution is increased by a factor of 8.

[0114] In one embodiment, such as Figure 3 As shown, the preset network structure search space includes L network layers connected in sequence; each network layer includes multiple cells, which can be understood as... Figure 3 Each circle in the diagram represents a cell, and each column of cells represents a network layer. Each network layer can include four optional upsampling factors, namely {1, 2, 4, 8}. For ease of understanding, let's take the third network layer as an example. For cell i in the third network layer, the associated cells with path constraints in the previous network layer (i.e., the second network layer) are q1, q2, and q3, and the associated cells with path constraints in the next network layer (i.e., the fourth network layer) are j1, j2, and j3. For cell i in the third network layer, the computer device can determine the output features of the associated cells q1, q2, and q3 with path constraints in the second network layer, and based on the path weight factors between cell i and each associated cell q1, q2, and q3, fuse the output features of each associated cell q1, q2, and q3 to obtain the input features of cell i. The computer device can perform corresponding feature processing on the input features of cell i to obtain the output features of cell i, and input the output features of cell i into the associated cells j1, j2 and j3 in the 4th network layer that have path constraint relationships with cell i. By continuously iterating until the search stopping condition is met, the searched network structure can be obtained. It can be understood that determining the network structure requires determining a cell in each network layer as the basic building block of the network structure. The path formed by the selected cells in each layer is the network structure.

[0115] In the above embodiments, multiple sample images can be used to perform multiple rounds of iterative processing in a preset network structure search space. During each iteration, the path weight factors between cells can be updated based on the output results of each round. When the search stopping condition is met, the optimal network structure can be obtained. Based on the optimal network structure, an optimal initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network can be constructed. Therefore, by using the optimal initial upsampling network, the initial prior feature extraction network corresponding to each key information representation method, and the initial feature fusion network to restore the image quality, the quality of the restored image can be improved.

[0116] In one embodiment, each cell includes multiple sequentially connected feature processing blocks; performing corresponding feature processing on the input features of the cell to obtain the output features of the cell includes: determining the operation weight factors of the candidate operations corresponding to each feature processing block in the cell; determining the input features of each feature processing block based on the input features of the cell, and executing each candidate operation on the input features of the feature processing blocks respectively to obtain the output features of the feature processing blocks corresponding to each candidate operation; fusing the output features of each feature processing block corresponding to each candidate operation according to the operation weight factors of each candidate operation to obtain the output features of the feature processing block; fusing the output features of each feature processing block in the cell to obtain the output features of the cell; adjusting the weight factors of each path based on the output results of the current round, and taking the next round as the current round, and continuing the iteration, including: adjusting the weight factors of each path and each operation based on the output results of the current round, and taking the next round as the current round, and continuing the iteration.

[0117] In this context, a candidate operation is an operation that serves as a candidate and performs corresponding feature processing on the feature. The operation weight factor of a candidate operation is a parameter used to characterize the probability of performing corresponding feature processing on the feature through that candidate operation.

[0118] Specifically, the computer device can determine the operation weight factors of candidate operations corresponding to each feature processing block in the cell based on the output results of the previous training round. The computer device can determine the input features of each feature processing block based on the input features of the cell, and execute each candidate operation on the input features of the feature processing blocks respectively, obtaining the output features of each feature processing block corresponding to each candidate operation. The computer device can fuse the output features of each feature processing block corresponding to each candidate operation according to the operation weight factors of each candidate operation to obtain the output features of the feature processing block. The computer device can fuse the output features of each feature processing block in the cell to obtain the output features of the cell. The computer device can adjust the weight factors of each path based on the output results of the current round, and take the next round as the current round to continue iterating. This includes: adjusting the weight factors of each path and each operation based on the output results of the current round, taking the next round as the current round, and returning to the current iteration. Based on multiple sample images, the current network layer is determined from multiple network layers. For each cell in the current network layer, the output features of the associated cells that have path constraint relationships in at least some of the preceding network layers are determined. This process is iteratively executed until the search stopping condition is met, resulting in the searched network structure. Based on the searched network structure, an initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network are constructed.

[0119] In one embodiment, the computer device may directly use the input features of a cell as the input features of each feature processing block.

[0120] In one embodiment, the candidate operation may specifically include at least one of the following feature operations: 3*3 convolution, 5*5 convolution, 7*7 convolution, and jump connection.

[0121] In the above embodiments, during each iteration, the path weight factors between cells and the operation weight factors of each candidate operation can be updated based on the output results of each iteration. When the search stopping condition is met, the optimal network structure can be obtained. Based on the optimal network structure, an optimal initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network can be constructed. Thus, by using the optimal initial upsampling network, the initial prior feature extraction network corresponding to each key information representation method, and the initial feature fusion network to restore the image quality, the quality of the restored image can be improved.

[0122] In one embodiment, determining the input features of each feature processing block based on the input features of the cell includes: for each feature processing block in the cell, taking the input features of the cell and the output features of the preceding feature processing block in the cell as the input features of the feature processing block; the preceding feature processing block is the feature processing block that precedes the previous feature processing block in the various feature processing blocks of the cell.

[0123] Specifically, for each feature processing block in a cell, the computer device can use the cell's input features and the output features of the preceding feature processing blocks in the cell together as the input features of the feature processing block.

[0124] In one embodiment, such as Figure 4 As shown, the input features of cell 403 include the output features of cell 401 in the layer above cell 403 and the output features of cell 402 in the layer above cell 401. Cell 403 includes feature processing block 1, feature processing block 2, and feature processing block 3. The computer device can concatenate the output features of feature processing block 1, feature processing block 2, and feature processing block 3 in cell 403 to obtain the output features of cell 403. For each feature processing block in cell 403, the input features of cell 403 and the output features of the preceding feature processing block in cell 403 are used as the input features of that feature processing block. For ease of understanding, we will take feature processing block 2 in cell 403 as an example. The input features of feature processing block 2 in cell 403 include the input features of cell 403 (i.e., the output features ② of cell 401 and the output features ③ of cell 402) and the output features of the prior feature processing block in cell 403 (i.e., the output features ① of feature processing block 1).

[0125] In the above embodiments, by using the input features of the cell and the output features of the prior feature processing blocks in the cell as the input features of the feature processing block, the input features of the feature processing block can be made richer, thereby ensuring that the output features of the cell to which the feature processing block belongs are more accurate.

[0126] In one embodiment, the input features of a cell may include the output features of associated cells that have path constraint relationships in a preset number and preset number of prior network layers.

[0127] In one embodiment, the input features of a cell may include the output features of related cells in the previous network layer that have path constraints, and the output features of related cells in the layer above that previous network layer that have path constraints. The output features of a cell may be the concatenation of the output features of each feature processing block within the cell. It is understood that this embodiment is merely illustrative and does not specifically limit the number of preceding cells into which the input features are fed. The preceding cells are cells in the network layer preceding the cell's current network layer.

[0128] In one embodiment, the output characteristics of a cell can be represented by the following formula:

[0129]

[0130] Where Cell represents a cell, H l H represents the output characteristics of the first-level cell. l-1 H represents the output characteristics of cells in layer l-1. l-2 This indicates the output characteristics of cells at levels l-2. `Concat` represents concatenated functions, and `B` represents the total number of feature processing blocks in the cell. This represents the output feature of the i-th feature processing block in the first-level cell.

[0131] In one embodiment, for each feature processing block in a cell, the output features of the associated cells in the previous layer of the network, the output features of the associated cells in the layer above that layer, and the output features of all preceding feature processing blocks in the cell are used as the input features of the feature processing block.

[0132] In one embodiment, the input features of each feature processing block in a cell can be represented by the following formula:

[0133]

[0134] in, This represents the input feature of the i-th feature processing block in the first-level cell.

[0135] In one embodiment, the output feature of each feature processing block in a cell can be represented by the following formula:

[0136]

[0137] Where, x j This represents the j-th input feature of the i-th feature processing block, O j→i (x j) indicates that a candidate operation is performed on the j-th input feature of the i-th feature processing block.

[0138] In one embodiment, O can be j→i The following definitions are made:

[0139]

[0140] in, Let o(x) represent the set of operations including each candidate operation o. j This indicates that the candidate operation o is performed on the j-th input feature of the i-th feature processing block. This represents the operation weight factor for candidate operation o.

[0141] In one embodiment, during the network structure search process, for each cell in each network layer, the input feature of that cell may be the output feature of a related cell in the previous network layer with the same resolution, or the output feature of a related cell with a resolution twice as high as that cell, or the output feature of a related cell with a resolution twice as low as that cell.

[0142] In the network structure search process, the output features of each cell can be described as follows through forward propagation:

[0143]

[0144] Where s represents the upsampling factor, a represents the hyperparameter of each cell (which can be manually set), and β represents the path weight factor between the cell and its associated cells. The path weight factor β can be normalized as:

[0145]

[0146] in, This means that the normalization formula applies to any s and any l.

[0147] In one embodiment, the target image is obtained by outputting a trained image quality restoration model; the step of obtaining the trained image quality restoration model includes: acquiring sample images; the sample images include a first sample image and a second sample image with the same content as the first sample image; the image quality of the second sample image is higher than that of the first sample image; the first sample image includes a first single-factor sample image and a first multi-factor sample image; the first single-factor sample image is a first sample image generated based on multiple different single factors; the first multi-factor sample image is a first sample image generated based on multiple different single factors; training the image quality restoration model to be trained using the sample images to obtain the trained image quality restoration model; the trained image quality restoration model includes a prior feature extraction network corresponding to each key information representation method.

[0148] It can be understood that the first sample image is the image to be restored during the training process of the image quality restoration model. The second sample image is the image to be restored during the training process of the image quality restoration model. It can be understood that we hope the quality of the image restored from the first sample image through the image quality restoration model will be close to the image quality of the second sample image. Single factor refers to a single factor causing low image quality. Multi-factor refers to multiple single factors jointly causing low image quality.

[0149] Specifically, the computer device can acquire sample images and iteratively train the image quality restoration model to be trained using the acquired sample images. That is, it continuously updates the model parameters of the image quality restoration model until the iteration stops, thus obtaining the trained image quality restoration model. The trained image quality restoration model includes a prior feature extraction network corresponding to each key information representation method.

[0150] In one embodiment, the iteration stopping condition can be that the number of iterations reaches a preset number of iterations, or that the model parameters of the image quality restoration model are no longer updated.

[0151] In one embodiment, factors that cause poor image quality may specifically include at least one of blur, noise, low resolution, and compression.

[0152] In one embodiment, a computer device can extract multi-factor features from a first multi-factor sample image. The multi-factor features can be extracted using the following formula:

[0153] Z full =F φ (I full )

[0154] Among them, Z full Indicating multifactor characteristics, Ifull F represents the first multi-factor sample image. φ () represents a multi-factor feature extraction network.

[0155] In one embodiment, the first multi-factor sample image can be generated jointly by multiple different single-factor features. It can be understood that the multi-factor features in the first multi-factor sample image can be obtained by superimposing multiple different single-factor features, and the multi-factor features can be represented by the following formula:

[0156] Z full ≈Z blur +Z noise +Z lr +Z jpeg

[0157] Among them, Z blur Z represents the characteristics of fuzzy factors. noise Z represents the characteristics of noise factors. lr Z represents the low-resolution factor characteristics. jpeg This indicates the characteristics of compression factors.

[0158] In one embodiment, the image quality restoration model to be trained learns single-factor features from a first single-factor sample image and multi-factor features from a first multi-factor sample image. The process of the image quality restoration model learning features can be described by the following formula:

[0159] Z full Z single =F φ (I full I single )

[0160] Among them, I single Z represents the image of each first single-factor sample. single This represents the characteristics of each individual factor.

[0161] It is understood that the image quality restoration model to be trained in this application can learn multi-factor features and various single-factor features simultaneously during the training process.

[0162] In the above embodiments, by training the image quality restoration model to be trained using the first single-factor sample image and the first multi-factor sample image, the image quality restoration model can learn the features of each single factor that causes low image quality, as well as the features of the multi-factor that causes low image quality, which are obtained by superimposing multiple factors. This makes the trained image quality restoration model widely applicable to the image quality restoration of low-quality images caused by various factors.

[0163] In one embodiment, training an image quality restoration model to be trained using sample images to obtain a trained image quality restoration model includes: inputting a first sample image into the image quality restoration model to be trained to restore the quality of the first sample image using the image quality restoration model to be trained, thereby obtaining a restored image; determining a target loss value based on the difference between the restored image and a second sample image; and iteratively training the image quality restoration model to be trained in the direction that reduces the target loss value until the iteration stops, thereby obtaining a trained image quality restoration model.

[0164] The restored image refers to the image obtained by restoring the image quality of the first sample image used as training data during the model training process of the image quality restoration model. The target loss value is the loss value used as the target during the model training process of the image quality restoration model.

[0165] Specifically, the computer device can input a first sample image into the image quality restoration model to be trained, so that the image quality of the first sample image can be restored by the image quality restoration model to obtain a restored image. The computer device can determine the difference between the restored image and the second sample image, and determine the target loss value based on the difference between the restored image and the second sample image. The computer device can iteratively train the image quality restoration model to be trained in the direction of reducing the target loss value until the iteration stopping condition is reached, thus obtaining the trained image quality restoration model.

[0166] In one embodiment, the computer device can determine the pixel differences between the recovered image and the second sample image, and directly determine the target loss function based on the pixel differences between the recovered image and the second sample image.

[0167] In one embodiment, a computer device can determine the feature differences between the features of the recovered image and the features of the second sample image, and directly determine the target loss function based on the feature differences between the features of the recovered image and the features of the second sample image.

[0168] In the above embodiments, the target loss value is determined by the difference between the recovered image and the second sample image, thereby enabling the training of the image quality restoration model. This can improve the image quality restoration effect of the trained image quality restoration model and thus obtain a better quality image.

[0169] In one embodiment, determining a target loss value based on the difference between the recovered image and the second sample image includes: determining a first loss value based on the pixel difference between the recovered image and the second sample image; extracting a first feature from the recovered image and a second feature from the second sample image, respectively; determining a second loss value based on the feature difference between the first feature and the second feature; and determining a target loss value based on the first loss value and the second loss value.

[0170] The first loss value is determined based on the pixel differences between the restored image and the second sample image. The second loss value is determined based on the feature differences between the features of the restored image and the features of the second sample image. The first feature is the feature extracted from the restored image. The second feature is the feature extracted from the second sample image.

[0171] Specifically, the computer device can determine the pixel differences between the recovered image and the second sample image, and determine a first loss value based on these pixel differences. The computer device can extract a first feature from the recovered image and a second feature from the second sample image, and determine the feature differences between the first and second features. Furthermore, the computer device can determine a second loss value based on the feature differences between the first and second features, and determine a target loss value based on both the first and second loss values.

[0172] In one embodiment, the target loss value can be determined by a target loss function, which is as follows:

[0173]

[0174] in, Indicates the target loss value. Indicates the second sample image, I HQ Let λ represent the recovered image, and φ() represent the pre-trained feature extraction network. The pre-trained feature extraction network φ() can extract the first feature of the recovered image and the second feature of the second sample image. per This represents the pre-set loss weight.

[0175] In the above embodiments, a target loss value is determined by a first loss value based on the pixel difference between the restored image and the second sample image, and a second loss value based on the feature difference between the first feature and the second feature. The image quality restoration model is then trained based on the target loss value, which can improve the image quality restoration effect of the trained image quality restoration model, thereby obtaining a better quality image and further improving the quality of the restored image.

[0176] In one embodiment, the target image is obtained by outputting a trained image quality restoration model; the image quality restoration model includes an initial feature extraction network; the initial feature extraction network includes convolutional layers and residual layers; the residual layer includes at least one residual fully connected unit; feature extraction of the first image to be restored to obtain the initial features of the first image includes: inputting the first image to be restored into the convolutional layer to convolve the first image to be restored through the convolutional layer to obtain convolutional features; inputting the convolutional features into the residual layer to process the convolutional features through at least one residual fully connected unit in the residual layer to obtain the initial features of the first image.

[0177] The initial feature extraction network is used to extract initial features from the first image to be restored. The residual fully connected unit includes multiple convolutional layers and activation layers between them. Each convolutional layer in each residual fully connected unit is connected to all the convolutional layers that follow it.

[0178] Specifically, the computer device can acquire a first image to be restored and input it into a convolutional layer to convolve the first image to be restored, thereby obtaining convolutional features. Then, the computer device can input the convolutional features into a residual layer, where at least one residual fully connected unit processes the convolutional features to obtain the initial features of the first image.

[0179] In the above embodiments, convolution is performed on the first image to be restored by a convolutional layer to obtain convolutional features. Then, at least one residual fully connected unit in the residual layer is used to process the convolutional features to obtain richer initial features of the first image. Based on the initial features of the first image, subsequent image quality restoration can be achieved, thereby improving the quality of the restored image.

[0180] In one embodiment, such as Figure 5As shown, the image quality restoration model may include an initial feature extraction network 502, an upsampling network 503 obtained through network structure search, a segmentation network 504a1, a heatmap generation network 504b1, a dictionary information extraction network 504c1, prior feature extraction networks (i.e., prior feature extraction networks 504a2, 504b2, and 504c2) obtained through network structure search and corresponding to each key information representation method, a feature fusion network 505 obtained through network structure search, a pixel reconstruction processing layer that performs feature processing on the fused features output by the feature fusion network, three convolutional layers, three residual units, and three upconvolutional layers. The initial feature extraction network 502 includes convolutional layers and residual layers. The residual layers include three residual fully connected units, each of which includes five convolutional layers a and four activation layers b. Each of these five convolutional layers a is connected to the convolutional layer a that follows it.

[0181] Reference Figure 5 During the training of the image quality restoration model, the computer device can train the image quality restoration model using a first multi-factor sample image 501a and multiple first single-factor sample images 501b.

[0182] Similarly, refer to Figure 5During the training and practical application of the image quality restoration model, the computer device can acquire the first image to be restored and input it into the initial feature extraction network 502. Features are extracted through the convolutional and residual layers of the initial feature extraction network 502 to obtain the initial features of the first image. The computer device can then input the initial features of the first image into the upsampling network 503 to perform upsampling processing on the initial features, obtaining upsampled features. Based on the upsampled features, the computer device can reconstruct a second image 501c that contains the image content of the first image and has a higher resolution than the first image. The computer device can obtain the key region segmentation map of the second image 501c under the first representation method through the segmentation network 504a1, obtain the key region heatmap under the second representation method through the heatmap generation network 504b1, and obtain the key component dictionary information under the third representation method through the dictionary information extraction network 504c1. The computer device can extract features of key region labeling information under the first representation method through the prior feature extraction network 504a2 corresponding to the first representation method, obtaining the prior features of key regions corresponding to the first representation method. It can also extract features of key region labeling information under the second representation method through the prior feature extraction network 504b2 corresponding to the second representation method, obtaining the prior features of key regions corresponding to the second representation method. Finally, it can extract features of key region labeling information under the third representation method through the prior feature extraction network 504c2 corresponding to the third representation method, obtaining the prior features of key regions corresponding to the third representation method. The computer device can then fuse the upsampled features and the prior features of key regions corresponding to the first, second, and third representation methods through the feature fusion network 505, obtaining fused features. Furthermore, the computer device can reconstruct the image using the fused features through a pixel reconstruction processing layer, three convolutional layers, three residual units, and three upconvolutional layers, obtaining the target image 506 restored from the first image.

[0183] like Figure 6 As shown, in one embodiment, an image processing method is provided. This method can be applied to a computer device, which can be a terminal or a server, and is executed independently by the terminal or server. It is understood that the method of this application can also be implemented through interaction between a terminal and a server. The method specifically includes the following steps:

[0184] Step 602: Obtain multiple sample images; the sample images include a first sample image and a second sample image with the same content as the first sample image; the image quality of the second sample image is higher than that of the first sample image; the first sample image includes a first single-factor sample image and a first multi-factor sample image; the first single-factor sample image is a first sample image generated based on multiple different single factors respectively; the first multi-factor sample image is a first sample image generated based on multiple different single factors together.

[0185] Step 604: In this iteration, based on multiple sample images, the current network layer is determined from multiple network layers in the preset network structure search space; the preset network structure search space includes multiple network layers connected in sequence; each network layer includes multiple cells; each cell includes multiple feature processing blocks connected in sequence.

[0186] Step 606: For each cell in the current network layer, determine the output features of the associated cells that have path constraint relationships in at least some of the previous network layers.

[0187] Step 608: Based on the path weight factors between the cell and each associated cell, the output features of each associated cell are fused to obtain the input features of the cell.

[0188] Step 610: Determine the operation weight factor for the candidate operation corresponding to each feature processing block in the cell.

[0189] Step 612: For each feature processing block in the cell, use the cell's input features and the output features of the previous feature processing block in the cell as the input features of the feature processing block.

[0190] Step 614: Perform each candidate operation on the input features of the feature processing block to obtain the output features of the feature processing block corresponding to each candidate operation.

[0191] Step 616: The output features of each feature processing block corresponding to each candidate operation are fused according to the operation weight factor of each candidate operation to obtain the output features of the feature processing block.

[0192] Step 618: Fuse the output features of each feature processing block in the cell to obtain the output features of the cell, and input the output features of the cell into the associated cells in the next layer of the network that have path constraints with the cell.

[0193] Step 620: Take the next network layer as the current network layer, return the step of determining the output features of the associated cells that have path constraint relationships in at least some of the previous network layers for each cell in the current network layer, and continue to execute until each network layer has been traversed in this round and the output result of this round is obtained.

[0194] Step 622: Adjust the weight factors of each path and each operation based on the output results of this round, and take the next round as this round to continue iterating until the search stopping condition is met, and obtain the searched network structure.

[0195] Step 624: Based on the searched network structure, construct an initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network.

[0196] Step 626: Update the network parameters of the initial upsampling network, each initial prior feature extraction network, and the initial feature fusion network using multiple sample images to obtain the upsampling network, the prior feature extraction network corresponding to each key information representation method, and the feature fusion network.

[0197] Step 628: Extract features from the first image to be restored to obtain the initial features of the first image.

[0198] Step 630: Upsample the initial features by using an upsampled network obtained through network structure search and matching the first image to obtain upsampled features.

[0199] Step 632: Reconstruct a second image based on the upsampling features, which has the image content of the first image and a higher resolution than the first image; obtain the key region marking information of the second image under various key information representation methods.

[0200] In one embodiment, the key region identification information under multiple key information representation methods includes at least two of the following: a key region segmentation map under a first representation method, a key region heat map under a second representation method, and key component dictionary information under a third representation method.

[0201] Step 634: For each key information representation method, extract the features of key region labeling information under the key information representation method through the prior feature extraction network corresponding to the key information representation method and obtained by network structure search processing, and obtain the key region prior features corresponding to the key information representation method.

[0202] Step 636: The upsampled features and the prior features of key regions corresponding to various key information representation methods are fused through the feature fusion network obtained by network structure search and matching the first image to obtain the fused features.

[0203] Step 638: Reconstruct the image based on the fusion features to obtain the target image restored from the first image.

[0204] This application also provides an application scenario in which the above-described image processing method is applied. Specifically, the image processing method can be applied to the scenario of face image quality restoration. A computer device can acquire multiple sample face images; the sample face images include a first sample face image and a second sample face image with the same content as the first sample face image; the face image quality of the second sample face image is higher than that of the first sample face image; the first sample face image includes a first single-factor sample face image and a first multi-factor sample face image; the first single-factor sample face image is a first sample face image generated based on multiple different single factors; the first multi-factor sample face image is a first sample face image generated based on multiple different single factors. In this iteration, based on multiple sample face images, the current network layer is determined from multiple network layers in a preset network structure search space; the preset network structure search space includes multiple network layers connected in sequence; each network layer includes multiple cells; each cell includes multiple feature processing blocks connected in sequence. For each cell in the current network layer, the output features of associated cells that have path constraint relationships in at least some of the preceding network layers are determined. Based on the path weight factors between a cell and its associated cells, the output features of each associated cell are fused to obtain the cell's input features. The operation weight factors for candidate operations corresponding to each feature processing block in the cell are determined. For each feature processing block in the cell, the cell's input features and the output features of its preceding feature processing blocks within the cell are used as the input features of the feature processing block. Each candidate operation is executed on the input features of the feature processing block to obtain the output features corresponding to each candidate operation. The output features of each feature processing block corresponding to each candidate operation are fused according to the operation weight factors of each candidate operation to obtain the output features of the feature processing block. The output features of each feature processing block in the cell are fused to obtain the cell's output features, and these output features are input to the associated cells in the next layer of the network that have path constraints with the cell. The next layer of the network is used as the current layer, and the step of determining the output features of associated cells that have path constraints with the cell in at least some preceding layers for each cell in the current layer is returned to continue execution until each network layer has been traversed in this round, yielding the output result for this round. Based on the output of this round, adjust the weight factors of each path and each operation, and take the next round as this round to continue iterating until the search stopping condition is met, and obtain the searched network structure.

[0205] The computer device can construct an initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network based on the searched network structure. The network parameters of the initial upsampling network, each initial prior feature extraction network, and the initial feature fusion network are updated using multiple sample face images to obtain the upsampling network, the prior feature extraction network corresponding to each key information representation method, and the feature fusion network.

[0206] The computer device can extract features from the first face image to be restored, obtaining the initial features of the first face image. An upsampling network, obtained through network structure search and matched with the first face image, is used to upsample the initial features, resulting in upsampled features. Based on the upsampled features, a second face image with the face image content of the first face image and a higher resolution is reconstructed; key region marking information of the second face image under various key information representation methods is obtained. For each key information representation method, features of the key region marking information under that key information representation method are extracted through a prior feature extraction network, obtained through network structure search, resulting in prior features of the key regions corresponding to that key information representation method. A feature fusion network, obtained through network structure search and matched with the first face image, fuses the upsampled features and the prior features of the key regions corresponding to each key information representation method, resulting in fused features. Based on the fused features, face image reconstruction is performed to obtain the target face image restored from the first face image.

[0207] This application also provides another application scenario in which the above-described image processing method is applied. Specifically, the image processing method can be applied to scenarios involving the restoration of animal image quality, plant image quality, human image quality, landscape image quality, and building image quality.

[0208] For example, in the scenario of animal image quality restoration, using the image processing method of this application, a computer device can extract features from a first animal image to be restored, obtaining initial features of the first animal image. These initial features are then subjected to resolution enhancement processing to obtain processed features. A second animal image is constructed based on the initial features, and the processed features, along with prior features of key regions of the second animal image corresponding to various key information representation methods, are fused to obtain fused features. Furthermore, based on these fused features, a target animal image with higher image quality than the first animal image can be reconstructed. This improves the image quality of the target animal image after image quality restoration.

[0209] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially, these steps are not necessarily executed in that order. Unless otherwise expressly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the above embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0210] In one embodiment, such as Figure 7 As shown, an image processing apparatus 700 is provided. This apparatus can be a software module, a hardware module, or a combination of both, integrated into a computer device. Specifically, the apparatus includes:

[0211] The extraction module 702 is used to extract features from the first image to be restored, so as to obtain the initial features of the first image;

[0212] The reconstruction module 704 is used to perform resolution enhancement processing on the initial features and reconstruct a second image with the image content of the first image and a resolution higher than that of the first image based on the processed features.

[0213] The acquisition module 706 is used to acquire key region marking information of the second image under multiple key information representation methods;

[0214] The extraction module 702 is also used to extract the features of the key region marking information under the key information representation method by using the prior feature extraction network corresponding to the key information representation method and obtained by network structure search processing for each key information representation method, so as to obtain the key region prior features corresponding to the key information representation method.

[0215] The fusion module 708 is used to fuse the processed features and the prior features of key regions corresponding to various key information representation methods to obtain fused features.

[0216] The reconstruction module 704 is also used to reconstruct the image based on the fusion features to obtain the target image restored from the first image.

[0217] In one embodiment, the reconstruction module 704 is further configured to perform upsampling processing on the initial features through an upsampling network obtained by network structure search and matching the first image, to obtain upsampled features; and reconstruct a second image with the image content of the first image and a resolution higher than that of the first image based on the upsampled features.

[0218] In one embodiment, the fusion module 708 is further configured to fuse the upsampled features and the prior features of key regions corresponding to various key information representation methods through a feature fusion network obtained by network structure search and matching the first image, to obtain fused features.

[0219] In one embodiment, the key region identification information under multiple key information representation methods includes at least two of the following: a key region segmentation map under a first representation method, a key region heat map under a second representation method, and key component dictionary information under a third representation method.

[0220] In one embodiment, the prior feature extraction network corresponding to each key information representation method belongs to a sub-network of the trained image quality restoration model; the trained image quality restoration model also includes an upsampling network and a feature fusion network; the resolution enhancement processing is performed by the upsampling network; the fused features are obtained by the feature fusion network. (Reference) Figure 8 The image processing apparatus 700 may further include:

[0221] The training module 710 is used to acquire multiple sample images; the image quality of the multiple sample images is similar to that of the first image; based on the multiple sample images, a network structure search is performed in a preset network structure search space, and an initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network are constructed based on the searched network structure; the network parameters of the initial upsampling network, each initial prior feature extraction network, and the initial feature fusion network are updated using multiple sample images to obtain the upsampling network, the prior feature extraction network corresponding to each key information representation method, and the feature fusion network.

[0222] In one embodiment, the preset network structure search space includes multiple network layers connected in sequence; each network layer includes multiple cells; the training module 710 is further configured to, in this iteration, determine the current network layer from the multiple network layers based on multiple sample images, and for each cell in the current network layer, determine the output features of associated cells that have path constraint relationships in at least some of the preceding network layers, and fuse the output features of each associated cell based on the path weight factors between the cell and each associated cell to obtain the input features of the cell; perform corresponding feature processing on the input features of the cell to obtain the output features of the cell, and pass the output features of the cell to the next network layer. The process involves taking the input of associated cells with path constraints, using the next network layer as the current network layer, and returning the steps for determining the output features of associated cells with path constraints in at least some of the preceding network layers for each cell in the current network layer. This process continues until each network layer has been traversed in this round, yielding the output result of this round. Based on the output result of this round, the path weight factors are adjusted, and the next round is used as this round to continue iterating until the search stopping condition is met, resulting in the searched network structure. Based on the searched network structure, an initial upsampling network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network are constructed.

[0223] In one embodiment, each cell includes multiple feature processing blocks connected in sequence; the training module 710 is further configured to determine the operation weight factors of the candidate operations corresponding to each feature processing block in the cell; determine the input features of each feature processing block based on the input features of the cell, and execute each candidate operation on the input features of the feature processing blocks respectively to obtain the output features of each feature processing block corresponding to each candidate operation; fuse the output features of each feature processing block corresponding to each candidate operation according to the operation weight factors of each candidate operation to obtain the output features of the feature processing block; fuse the output features of each feature processing block in the cell to obtain the output features of the cell; adjust the weight factors of each path and each operation based on the output results of this round, and take the next round as this round to continue the iteration.

[0224] In one embodiment, the training module 710 is further configured to, for each feature processing block in a cell, take the input features of the cell and the output features of the preceding feature processing block in the cell as the input features of the feature processing block; the preceding feature processing block is the feature processing block that precedes the previous feature processing block among the various feature processing blocks in the cell.

[0225] In one embodiment, the target image is obtained by outputting a trained image quality restoration model; the training module 710 is also used to acquire sample images; the sample images include a first sample image and a second sample image with the same content as the first sample image; the image quality of the second sample image is higher than that of the first sample image; the first sample image includes a first single-factor sample image and a first multi-factor sample image; the first single-factor sample image is a first sample image generated based on multiple different single factors; the first multi-factor sample image is a first sample image generated based on multiple different single factors; the image quality restoration model to be trained is trained using the sample images to obtain a trained image quality restoration model; the trained image quality restoration model includes a prior feature extraction network corresponding to each key information representation method.

[0226] In one embodiment, the training module 710 is further configured to input the first sample image into the image quality restoration model to be trained, so as to restore the quality of the first sample image through the image quality restoration model to be trained, and obtain the restored image; determine the target loss value based on the difference between the restored image and the second sample image; and iteratively train the image quality restoration model to be trained in the direction of reducing the target loss value until the iteration stopping condition is reached, so as to obtain the trained image quality restoration model.

[0227] In one embodiment, the training module 710 is further configured to determine a first loss value based on the pixel difference between the recovered image and the second sample image; extract a first feature of the recovered image and a second feature of the second sample image respectively; determine a second loss value based on the feature difference between the first feature and the second feature; and determine a target loss value based on the first loss value and the second loss value.

[0228] In one embodiment, the target image is obtained by outputting a trained image quality restoration model; the image quality restoration model includes an initial feature extraction network; the initial feature extraction network includes a convolutional layer and a residual layer; the residual layer includes at least one residual fully connected unit; the extraction module 702 is further configured to input the first image to be restored into the convolutional layer, so as to convolve the first image to be restored through the convolutional layer to obtain convolutional features; and input the convolutional features into the residual layer, so as to process the convolutional features through at least one residual fully connected unit in the residual layer to obtain the initial features of the first image.

[0229] The aforementioned image processing device extracts features from the first image to be restored, obtaining initial features of the first image. It then performs resolution enhancement processing on these initial features to obtain processed features with higher resolution. Based on these processed features, a second image containing the image content of the first image and with higher resolution can be reconstructed. The device acquires key region marking information of the second image under various key information representation methods. For each key information representation method, a prior feature extraction network corresponding to that method, obtained through network structure search processing, extracts features of the key region marking information under that key information representation method, obtaining prior features of the key regions corresponding to that key information representation method. Furthermore, it fuses the processed features with the prior features of the key regions corresponding to the various key information representation methods to obtain fused features. Based on these fused features, image reconstruction yields a target image restored from the first image, with image quality higher than both the first and second images. Compared to traditional manually designed networks, this application uses a prior feature extraction network obtained through network structure search processing corresponding to the key information representation method. This network can extract key region labeling information with better performance under the key information representation method. At the same time, by fusing the features after resolution enhancement and the key region prior features corresponding to various key information representation methods, richer fused features can be obtained. Based on these fused features, a target image with higher image quality can be reconstructed, thereby improving the quality of the recovered image.

[0230] Each module in the aforementioned image processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0231] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media to run. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements an image processing method.

[0232] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements an image processing method. The display unit of the computer device is used to form a visually visible image. It can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0233] Those skilled in the art will understand that Figure 9 and Figure 10The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0234] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0235] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0236] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0237] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0238] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0239] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0240] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Performing feature extraction on the first image to be restored to obtain initial features of the first image; The resolution enhancement network obtained by the network structure search process is used to perform resolution enhancement processing on the initial features, and a second image having the image content of the first image and a higher resolution than the first image is reconstructed based on the processed features; the network structure search process refers to a processing method of searching for an optimal network structure in a preset network structure search space, and constructing a corresponding neural network based on the searched network structure; the preset network structure search space includes a plurality of network layers connected in sequence; each of the network layers includes a plurality of cells; Acquire key area marking information of the second image in multiple key information representation modes; the key area marking information in multiple key information representation modes includes at least two of a key area segmentation map in a first representation mode, a key area heat map in a second representation mode, and key component dictionary information in a third representation mode; For each key information representation method, extract the features of the key area marking information under the key information representation method through a priori feature extraction network corresponding to the key information representation method and obtained by network structure search processing, and obtain the key area priori features corresponding to the key information representation method; The processed features and the key area prior features corresponding to the multiple key information representation methods are fused through a feature fusion network obtained by network structure search processing to obtain fused features; Reconstructing the image based on the fusion features to obtain a target image restored from the first image; The resolution enhancement network, the prior feature extraction network and the feature fusion network are obtained by the following steps: Acquire a plurality of sample images; the plurality of sample images have image quality similar to that of the first image; In this round of iteration, based on the multiple sample images, a current network layer is determined from the multiple network layers, and for each cell in the current network layer, output features of associated cells having a path constraint relationship with the cell in at least part of the previous network layer are determined, and based on the path weight factor between the cell and each associated cell, the output features of each associated cell are fused to obtain the input feature of the cell; Performing corresponding feature processing on the input feature of the cell to obtain the output feature of the cell, and inputting the output feature of the cell to an associated cell in a next network layer that has a path constraint relationship with the cell; Taking the next network layer as the current network layer, returning to the step of determining, for each cell in the current network layer, the output characteristics of the associated cells of the cell having a path constraint relationship in at least part of the previous network layer, and continuing to execute until each network layer is traversed in this round to obtain the output result of this round; Adjust the weight factors of each path based on the output results of the current round, and use the next round as the current round to continue iterating until the search stop condition is met, obtain the searched network structure, and construct an initial resolution enhancement network, an initial prior feature extraction network corresponding to each key information representation method, and an initial feature fusion network based on the searched network structure; The network parameters of the initial resolution enhancement network, each of the initial prior feature extraction networks, and the initial feature fusion network are updated through the multiple sample images to obtain the resolution enhancement network, the prior feature extraction network corresponding to each key information representation method, and the feature fusion network.

2. The method according to claim 1, characterized in that The resolution enhancement network includes an upsampling network; the resolution enhancement network obtained by the network structure search process performs resolution enhancement processing on the initial features, and reconstructs a second image having the image content of the first image and a higher resolution than the first image based on the processed features, including: Performing upsampling processing on the initial features through an upsampling network obtained by network structure search and matching the first image to obtain upsampled features; A second image having the image content of the first image and a higher resolution than the first image is reconstructed based on the up-sampled features.

3. The method according to claim 2, characterized in that The feature fusion network obtained by the network structure search process fuses the processed features with the key area prior features corresponding to the multiple key information representation methods to obtain fused features, including: The up-sampled features and the key area prior features corresponding to the multiple key information representation methods are fused through a feature fusion network obtained by network structure search processing and matched with the first image to obtain a fused feature.

4. The method according to claim 1, characterized in that: The prior feature extraction network corresponding to each key information representation method belongs to a sub-network of the trained image quality restoration model; the trained image quality restoration model also includes an upsampling network and a feature fusion network; the resolution enhancement processing is performed by the upsampling network; the fused feature is obtained by the feature fusion network.

5. The method according to claim 1, characterized in that Each of the cells includes a plurality of feature processing blocks connected in sequence; the corresponding feature processing is performed on the input features of the cell to obtain the output features of the cell, including: Determine an operation weight factor of a candidate operation corresponding to each feature processing block in the cell; Determine input features of each feature processing block based on the input features of the cell, and execute each candidate operation on the input features of the feature processing block to obtain output features of the feature processing block corresponding to each candidate operation; The output features of each feature processing block corresponding to each candidate operation are merged according to the operation weight factor of each candidate operation to obtain the output features of the feature processing block; The output features of each feature processing block in the cell are merged to obtain the output features of the cell; The step of adjusting each path weight factor based on the output result of the current round, and taking the next round as the current round to continue iterating includes: The path weight factors and the operation weight factors are adjusted based on the output results of the current round, and the next round is used as the current round to continue iterating.

6. The method according to claim 5, characterized in that The step of determining the input features of each feature processing block based on the input features of the cell includes: For each feature processing block in the cell, using the input feature of the cell and the output feature of the previous feature processing block of the feature processing block in the cell as the input feature of the feature processing block; The preceding feature processing block is a feature processing block located before the feature processing block among the feature processing blocks of the cell.

7. The method according to claim 1, characterized in that The target image is obtained by outputting a trained image quality restoration model; and the step of obtaining the trained image quality restoration model comprises: Acquire sample images; the sample images include a first sample image and a second sample image with the same content as the first sample image; the image quality of the second sample image is higher than that of the first sample image; the first sample image includes a first single-factor sample image and a first multi-factor sample image; the first single-factor sample image is a first sample image generated based on a plurality of different single factors respectively; the first multi-factor sample image is a first sample image generated based on the plurality of different single factors; The image quality restoration model to be trained is trained by using the sample images to obtain a trained image quality restoration model; the trained image quality restoration model includes a priori feature extraction network corresponding to each key information representation method.

8. The method according to claim 7, characterized in that The image quality restoration model to be trained is trained by using the sample image to obtain a trained image quality restoration model, including Inputting the first sample image into the image quality restoration model to be trained, so as to restore the quality of the first sample image through the image quality restoration model to be trained, and obtain a restored image; determining a target loss value based on a difference between the restored image and the second sample image; The image quality restoration model to be trained is iteratively trained in a direction that reduces the target loss value until an iteration stop condition is reached, thereby obtaining a trained image quality restoration model.

9. The method according to claim 8, characterized in that The determining of a target loss value based on a difference between the restored image and the second sample image comprises: determining a first loss value based on a pixel difference between the restored image and the second sample image; extracting a first feature of the restored image and a second feature of the second sample image respectively; determining a second loss value based on a feature difference between the first feature and the second feature; A target loss value is determined based on the first loss value and the second loss value.

10. The method according to any one of claims 1 to 9, characterized in that The target image is obtained by outputting a trained image quality restoration model; the image quality restoration model includes an initial feature extraction network; the initial feature extraction network includes a convolutional layer and a residual layer; the residual layer includes at least one residual fully connected unit; The step of extracting features from the first image to be restored to obtain initial features of the first image includes: Inputting the first image to be restored into the convolution layer, so as to convolve the first image to be restored through the convolution layer to obtain convolved features; The convolved features are input into the residual layer, so that the convolved features are processed by the at least one residual fully connected unit in the residual layer to obtain initial features of the first image.

11. An image processing device, characterized in that: The device comprises: An extraction module, used for performing feature extraction on the first image to be restored to obtain initial features of the first image; A reconstruction module, used to perform resolution enhancement processing on the initial features through a resolution enhancement network obtained by network structure search processing, and reconstruct a second image having the image content of the first image and a higher resolution than the first image based on the processed features; the network structure search processing refers to a processing method of searching for an optimal network structure in a preset network structure search space, and constructing a corresponding neural network based on the searched network structure; the preset network structure search space includes a plurality of network layers connected in sequence; each of the network layers includes a plurality of cells; an acquisition module, configured to acquire key area marking information of the second image in a plurality of key information representation modes; the key area marking information in a plurality of key information representation modes comprises at least two of a key area segmentation map in a first representation mode, a key area heat map in a second representation mode, and key component dictionary information in a third representation mode; The extraction module is also used to extract the features of the key area marking information under each key information representation mode through a priori feature extraction network corresponding to the key information representation mode and obtained by network structure search processing, so as to obtain the key area priori features corresponding to the key information representation mode; A fusion module, used to fuse the processed features and the key area prior features corresponding to the multiple key information representation methods through a feature fusion network obtained by network structure search processing, to obtain a fused feature; The reconstruction module is also used to reconstruct the image based on the fusion feature to obtain a target image restored from the first image; A training module is used to obtain multiple sample images; the multiple sample images have similar image quality to the first image; in this round of iteration, based on the multiple sample images, the current network layer is determined from the multiple network layers, and for each cell in the current network layer, the output features of the associated cells with which the cell has a path constraint relationship in at least part of the previous network layer are determined, and based on the path weight factor between the cell and the associated cells, the output features of the associated cells are fused to obtain the input features of the cell; the input features of the cell are processed accordingly to obtain the output features of the cell, and the output features of the cell are input to the associated cells with which the cell has a path constraint relationship in the next network layer; the next network layer is taken as the current network layer, and the output features of the cell determined for each cell in the current network layer are returned. The step of outputting features of associated cells having path constraint relationships in at least part of the previous network layer is continued until each network layer is traversed in this round to obtain the output result of this round; adjusting each path weight factor based on the output result of this round, and taking the next round as this round, and continuing to iterate until the search stop condition is met to obtain the searched network structure, and constructing an initial resolution improvement network, an initial priori feature extraction network corresponding to each key information representation method, and an initial feature fusion network based on the searched network structure; updating the network parameters of the initial resolution improvement network, each initial priori feature extraction network, and the initial feature fusion network through the multiple sample images to obtain the resolution improvement network, the priori feature extraction network corresponding to each key information representation method, and the feature fusion network.

12. The device according to claim 11, characterized in that The resolution enhancement network includes an upsampling network; the reconstruction module is also used to upsample the initial features through an upsampling network obtained by network structure search and matching the first image to obtain upsampled features; based on the upsampled features, a second image having the image content of the first image and a higher resolution than the first image is reconstructed.

13. The device according to claim 12, characterized in that The fusion module is also used to fuse the up-sampled features and the key area prior features corresponding to the multiple key information representation methods through a feature fusion network obtained by network structure search processing and matched with the first image, so as to obtain a fused feature.

14. The device according to claim 11, characterized in that The prior feature extraction network corresponding to each key information representation method belongs to a sub-network of the trained image quality restoration model; the trained image quality restoration model also includes an upsampling network and a feature fusion network; the resolution enhancement processing is performed by the upsampling network; the fused feature is obtained by the feature fusion network.

15. The device according to claim 11, characterized in that Each of the cells includes a plurality of feature processing blocks connected in sequence; the training module is further used to determine the operation weight factor of the candidate operation corresponding to each feature processing block in the cell; determine the input features of each feature processing block based on the input features of the cell, and execute each candidate operation on the input features of the feature processing block respectively to obtain the output features of the feature processing block corresponding to each candidate operation; fuse the output features of each feature processing block corresponding to each candidate operation according to the operation weight factor of each candidate operation to obtain the output features of the feature processing block; fuse the output features of each feature processing block in the cell to obtain the output features of the cell; The path weight factors and the operation weight factors are adjusted based on the output results of the current round, and the next round is used as the current round to continue iterating.

16. The device according to claim 15, characterized in that The training module is also used to use the input features of the cell and the output features of the previous feature processing block of the feature processing block in the cell as the input features of the feature processing block for each feature processing block in the cell; the previous feature processing block is a feature processing block that is located before the feature processing block among the feature processing blocks of the cell.

17. The device according to claim 11, characterized in that The target image is obtained by outputting a trained image quality restoration model; the training module is also used to obtain sample images; the sample images include a first sample image and a second sample image with the same content as the first sample image; the image quality of the second sample image is higher than that of the first sample image; the first sample image includes a first single-factor sample image and a first multi-factor sample image; the first single-factor sample image is a first sample image generated based on a plurality of different single factors respectively; the first multi-factor sample image is a first sample image generated based on the plurality of different single factors; The image quality restoration model to be trained is trained by using the sample images to obtain a trained image quality restoration model; the trained image quality restoration model includes a priori feature extraction network corresponding to each key information representation method.

18. The device according to claim 17, characterized in that The training module is further used for inputting the first sample image into the image quality restoration model to be trained, so as to restore the quality of the first sample image through the image quality restoration model to be trained to obtain a restored image; Based on the difference between the restored image and the second sample image, a target loss value is determined; and the image quality restoration model to be trained is iteratively trained in a direction that reduces the target loss value until an iteration stop condition is reached, thereby obtaining a trained image quality restoration model.

19. The device according to claim 18, characterized in that The training module is also used to determine a first loss value based on the pixel difference between the restored image and the second sample image; extract the first feature of the restored image and the second feature of the second sample image respectively; determine a second loss value based on the feature difference between the first feature and the second feature; and determine a target loss value based on the first loss value and the second loss value.

20. The device according to any one of claims 11 to 19, characterized in that The target image is obtained by outputting a trained image quality restoration model; the image quality restoration model includes an initial feature extraction network; the initial feature extraction network includes a convolutional layer and a residual layer; the residual layer includes at least one residual fully connected unit; the extraction module is also used to input the first image to be restored into the convolutional layer, so as to convolve the first image to be restored through the convolutional layer to obtain the convolved features; the convolved features are input into the residual layer, so as to process the convolved features through the at least one residual fully connected unit in the residual layer to obtain the initial features of the first image.

21. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 10 are implemented.

22. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

23. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.

Images