Image model processing method and device, electronic equipment, readable storage medium and program product

Abstract

The invention relates to an image model processing method and device, electronic equipment, a readable storage medium and a program product. The method comprises the steps of performing semantic segmentation on a first resolution image through a target semantic segmentation model, and obtaining at least two first intermediate semantic segmentation features of different scales; converting each first intermediate semantic segmentation feature into a first semantic correlation weight matrix; elements in the first semantic relevance weight matrix represent relevance between corresponding positions of the elements in the first intermediate semantic segmentation feature and other positions; performing super-division processing on the first resolution image based on each first semantic correlation weight matrix through a first super-division model to obtain a super-division image; and obtaining loss according to the super-resolution image and a reference image corresponding to the first resolution image, and adjusting the first super-resolution model according to the loss to obtain a target super-resolution model. By adopting the method, the target super-division model with more accurate super-division processing can be provided.

Description

Technical Field

[0001] The present application relates to the field of imaging technology, and in particular to an image model processing method and device, an image processing method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] In the field of image processing and computer vision, high-magnification image super-resolution is a very challenging task. Its goal is to reconstruct a high-resolution (HR) image through calculation based on a low-resolution (LR) image, thereby significantly improving the clarity and details of the image. This has important practical significance for applications in many fields such as satellite imaging, medical image analysis, and mobile phone side photography.

[0003] Usually, a super-resolution model is used to super-resolve low-resolution images to obtain high-resolution images. Therefore, the super-resolution model is the key. However, the traditional super-resolution model has the problem of inaccurate super-resolution processing. Summary of the invention

[0004] The embodiments of the present application provide an image model processing method and device, an image processing method and device, an electronic device, a computer-readable storage medium, and a computer program product, which can provide a target super-resolution model with more accurate super-resolution processing.

[0005] In a first aspect, the present application provides an image model processing method, comprising:

[0006] Performing semantic segmentation on the first resolution image through a target semantic segmentation model, and obtaining first intermediate semantic segmentation features of at least two different scales generated in the semantic segmentation process;

[0007] Converting each first intermediate semantic segmentation feature into a first semantic relevance weight matrix; the elements in the first semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature;

[0008] Injecting each first semantic relevance weight matrix into a first super-resolution model, and performing super-resolution processing on the first resolution image based on each first semantic relevance weight matrix through the first super-resolution model to obtain a super-resolution image; the resolution of the super-resolution image is greater than the resolution of the first resolution image;

[0009] A loss is obtained according to the super-resolution image and a reference image corresponding to the first resolution image, and the first super-resolution model is adjusted according to the loss to obtain a trained target super-resolution model.

[0010] In one embodiment, the step of performing super-resolution processing on the first resolution image based on each of the first semantic relevance weight matrices by the first super-resolution model to obtain a super-resolution image includes:

[0011] Performing super-resolution processing on the first resolution image by using the first super-resolution model to obtain a first intermediate super-resolution feature;

[0012] Obtain a first semantic relevance weight matrix of a scale corresponding to the first intermediate super-resolution feature;

[0013] Obtaining a second intermediate super-resolution feature based on the first intermediate super-resolution feature and the corresponding first semantic relevance weight matrix;

[0014] The second intermediate super-resolution feature is used as a new first intermediate super-resolution feature, and the step of obtaining a first semantic relevance weight matrix corresponding to a scale of the first intermediate super-resolution feature is returned to be executed until a super-resolution image is obtained.

[0015] In one embodiment, obtaining the second intermediate super-resolution feature based on the first intermediate super-resolution feature and the corresponding first semantic relevance weight matrix includes:

[0016] Convolving the first intermediate super-resolution feature to obtain a first intermediate super-resolution feature after convolution;

[0017] The first intermediate super-resolution feature after the convolution is matrix-multiplied with the corresponding first semantic relevance weight matrix to obtain an intermediate super-resolution feature of a second scale.

[0018] In one embodiment, converting each first intermediate semantic segmentation feature into a first semantic relevance weight matrix includes:

[0019] For each first intermediate semantic segmentation feature, convolve and upsample the first intermediate semantic segmentation feature to obtain a first intermediate semantic segmentation feature of a target size;

[0020] Performing a transposition process on the first intermediate semantic segmentation feature of the target size in terms of spatial dimension to obtain a transposed first intermediate semantic segmentation feature;

[0021] A first semantic relevance weight matrix is ​​obtained based on the first intermediate semantic segmentation features of the target size and the transposed first intermediate semantic segmentation features.

[0022] In one embodiment, the method further comprises:

[0023] Acquire a semantic segmentation result of the first resolution image by using the target semantic segmentation model;

[0024] The step of obtaining a loss according to the super-resolution image and a reference image corresponding to the first resolution image, and adjusting the first super-resolution model according to the loss to obtain a trained target super-resolution model includes:

[0025] Dividing the super-resolved image into super-resolved image blocks of at least two semantic categories based on the semantic segmentation result;

[0026] A loss is obtained according to the super-resolution image blocks of the at least two semantic categories and a reference image corresponding to the first resolution image, and the first super-resolution model is adjusted according to the loss to obtain a trained target super-resolution model.

[0027] In one embodiment, obtaining a loss based on the super-resolution image blocks of the at least two semantic categories and a reference image corresponding to the first resolution image, and adjusting the first super-resolution model based on the loss to obtain a trained target super-resolution model includes:

[0028] For each super-resolved image block of each semantic category, based on the super-resolved image block and a reference image corresponding to the first resolution image, determining a sub-discrimination result corresponding to the super-resolved image block;

[0029] Determining a discrimination result of the super-resolved image according to the sub-discrimination results corresponding to the super-resolved image blocks of each semantic category;

[0030] A loss is obtained based on the discrimination result, and the first super-resolution model is adjusted according to the loss to obtain a trained target super-resolution model.

[0031] In one embodiment, the method further comprises:

[0032] Get the semantic segmentation dataset;

[0033] Using the semantic segmentation dataset to train the initial semantic segmentation model to obtain a trained first candidate semantic segmentation model;

[0034] Acquire at least one second candidate semantic segmentation model different from the first candidate semantic segmentation model;

[0035] A target semantic segmentation model is determined from the first candidate semantic segmentation model and the at least one second candidate semantic segmentation model.

[0036] In a second aspect, the present application also provides an image model processing device, comprising:

[0037] A first semantic segmentation module, configured to perform semantic segmentation on the first resolution image by using a target semantic segmentation model, and obtain first intermediate semantic segmentation features of at least two different scales generated during the semantic segmentation process;

[0038] A first semantic relevance weight matrix acquisition module is used to convert each first intermediate semantic segmentation feature into a first semantic relevance weight matrix; the elements in the first semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature;

[0039] A first super-resolution processing module is used to inject each first semantic relevance weight matrix into a first super-resolution model, and perform super-resolution processing on the first resolution image based on each first semantic relevance weight matrix through the first super-resolution model to obtain a super-resolution image; the resolution of the super-resolution image is greater than the resolution of the first resolution image;

[0040] The target super-resolution model acquisition module is used to obtain the loss according to the super-resolution image and the reference image corresponding to the first resolution image, and adjust the first super-resolution model according to the loss to obtain the trained target super-resolution model.

[0041] In a third aspect, the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0042] Performing semantic segmentation on the first resolution image through a target semantic segmentation model, and obtaining first intermediate semantic segmentation features of at least two different scales generated in the semantic segmentation process;

[0043] Converting each first intermediate semantic segmentation feature into a first semantic relevance weight matrix; the elements in the first semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature;

[0044] Injecting each first semantic relevance weight matrix into a first super-resolution model, and performing super-resolution processing on the first resolution image based on each first semantic relevance weight matrix through the first super-resolution model to obtain a super-resolution image; the resolution of the super-resolution image is greater than the resolution of the first resolution image;

[0045] A loss is obtained according to the super-resolution image and a reference image corresponding to the first resolution image, and the first super-resolution model is adjusted according to the loss to obtain a trained target super-resolution model.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0047] Performing semantic segmentation on the first resolution image through a target semantic segmentation model, and obtaining first intermediate semantic segmentation features of at least two different scales generated in the semantic segmentation process;

[0048] Converting each first intermediate semantic segmentation feature into a first semantic relevance weight matrix; the elements in the first semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature;

[0049] Injecting each first semantic relevance weight matrix into a first super-resolution model, and performing super-resolution processing on the first resolution image based on each first semantic relevance weight matrix through the first super-resolution model to obtain a super-resolution image; the resolution of the super-resolution image is greater than the resolution of the first resolution image;

[0050] A loss is obtained according to the super-resolution image and a reference image corresponding to the first resolution image, and the first super-resolution model is adjusted according to the loss to obtain a trained target super-resolution model.

[0051] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0052] Performing semantic segmentation on the first resolution image through a target semantic segmentation model, and obtaining first intermediate semantic segmentation features of at least two different scales generated in the semantic segmentation process;

[0053] Converting each first intermediate semantic segmentation feature into a first semantic relevance weight matrix; the elements in the first semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature;

[0054] Injecting each first semantic relevance weight matrix into a first super-resolution model, and performing super-resolution processing on the first resolution image based on each first semantic relevance weight matrix through the first super-resolution model to obtain a super-resolution image; the resolution of the super-resolution image is greater than the resolution of the first resolution image;

[0055] A loss is obtained according to the super-resolution image and a reference image corresponding to the first resolution image, and the first super-resolution model is adjusted according to the loss to obtain a trained target super-resolution model.

[0056] The above-mentioned image model processing method, device, electronic device, computer-readable storage medium and computer program product perform semantic segmentation on the first resolution image through the target semantic segmentation model, and obtain at least two first intermediate semantic segmentation features of different scales generated in the semantic segmentation process, and convert each first intermediate semantic segmentation feature into a first semantic relevance weight matrix, the elements in the first semantic relevance weight matrix represent the correlation between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature. Then, each first semantic relevance weight matrix is ​​injected into the first super-resolution model, and the first super-resolution model can learn the correlation of the semantic positions in the first semantic relevance weight matrix, and perform feature consistency aggregation on pixels with consistent correlation, thereby ensuring that the texture distribution of the corresponding positions belongs to the same semantic category, and more accurately super-resolution processing is performed on the first resolution image, improving the details and consistency of the super-resolution processing in the super-resolution image, and better maintaining the semantic information and visual quality of the image, and then obtaining the loss according to the reference image corresponding to the super-resolution image and the first resolution image, and adjusting the first super-resolution model according to the loss to obtain a target super-resolution model with more accurate super-resolution processing.

[0057] In a sixth aspect, the present application provides an image processing method, comprising:

[0058] Performing semantic segmentation on the second resolution image through the target semantic segmentation model, and obtaining at least two second intermediate semantic segmentation features of different scales generated in the semantic segmentation process;

[0059] Converting each second intermediate semantic segmentation feature into a second semantic relevance weight matrix; the elements in the second semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the second intermediate semantic segmentation feature;

[0060] Each second semantic relevance weight matrix is ​​injected into a target super-resolution model, and the second resolution image is super-resolved based on each second semantic relevance weight matrix through the target super-resolution model to obtain a target image; the resolution of the target image is greater than the resolution of the second resolution image.

[0061] In a seventh aspect, the present application further provides an image processing device, comprising:

[0062] A second semantic segmentation module is used to perform semantic segmentation on the second resolution image by using a target semantic segmentation model, and obtain at least two second intermediate semantic segmentation features of different scales generated in the semantic segmentation process;

[0063] A second semantic relevance weight matrix acquisition module is used to convert each second intermediate semantic segmentation feature into a second semantic relevance weight matrix; the elements in the second semantic relevance weight matrix represent the correlation between the corresponding position of the element and other positions in the second intermediate semantic segmentation feature;

[0064] The second super-resolution processing module is used to inject each second semantic relevance weight matrix into the target super-resolution model, and perform super-resolution processing on the second resolution image based on the second semantic relevance weight matrix through the target super-resolution model to obtain a target image; the resolution of the target image is greater than the resolution of the second resolution image.

[0065] In an eighth aspect, the present application further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0066] Performing semantic segmentation on the second resolution image through the target semantic segmentation model, and obtaining at least two second intermediate semantic segmentation features of different scales generated in the semantic segmentation process;

[0067] Converting each second intermediate semantic segmentation feature into a second semantic relevance weight matrix; the elements in the second semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the second intermediate semantic segmentation feature;

[0068] Each second semantic relevance weight matrix is ​​injected into a target super-resolution model, and the second resolution image is super-resolved based on each second semantic relevance weight matrix through the target super-resolution model to obtain a target image; the resolution of the target image is greater than the resolution of the second resolution image.

[0069] In a ninth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0070] Performing semantic segmentation on the second resolution image through the target semantic segmentation model, and obtaining at least two second intermediate semantic segmentation features of different scales generated in the semantic segmentation process;

[0071] Converting each second intermediate semantic segmentation feature into a second semantic relevance weight matrix; the elements in the second semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the second intermediate semantic segmentation feature;

[0072] Each second semantic relevance weight matrix is ​​injected into a target super-resolution model, and the second resolution image is super-resolved based on each second semantic relevance weight matrix through the target super-resolution model to obtain a target image; the resolution of the target image is greater than the resolution of the second resolution image.

[0073] In a tenth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0074] Performing semantic segmentation on the second resolution image through the target semantic segmentation model, and obtaining at least two second intermediate semantic segmentation features of different scales generated in the semantic segmentation process;

[0075] Converting each second intermediate semantic segmentation feature into a second semantic relevance weight matrix; the elements in the second semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the second intermediate semantic segmentation feature;

[0076] Each second semantic relevance weight matrix is ​​injected into a target super-resolution model, and the second resolution image is super-resolved based on each second semantic relevance weight matrix through the target super-resolution model to obtain a target image; the resolution of the target image is greater than the resolution of the second resolution image.

[0077] The above-mentioned image processing method, device, electronic device, computer-readable storage medium and computer program product perform semantic segmentation on the second resolution image through the target semantic segmentation model, and obtain at least two second intermediate semantic segmentation features of different scales generated in the semantic segmentation process, and convert each second intermediate semantic segmentation feature into a second semantic relevance weight matrix, the elements in the second semantic relevance weight matrix represent the correlation between the corresponding position of the element and other positions in the second intermediate semantic segmentation feature. Then, each second semantic relevance weight matrix is ​​injected into the target super-resolution model. The target super-resolution model can learn the correlation of the semantic positions in the second semantic relevance weight matrix, and perform feature consistency aggregation on pixels with consistent correlation, thereby ensuring that the texture distribution of the corresponding positions belongs to the same semantic category, and more accurately super-resolution processing is performed on the second resolution image, improving the details and consistency of the super-resolution processing in the image, better maintaining the semantic information and visual quality of the image, and obtaining a target super-resolution image with more accurate super-resolution processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0079] Figure 1 is a flowchart of an image model processing method in one embodiment;

[0080] Figure 2 A schematic diagram of super-resolution processing in one embodiment;

[0081] Figure 3 A schematic diagram of determining a super-resolved image based on a semantic segmentation result in one embodiment;

[0082] Figure 4 is a flowchart of an image model processing method in another embodiment;

[0083] Figure 5 is a schematic flow chart of an image processing method in one embodiment;

[0084] Figure 6 is a structural block diagram of an image model processing device in one embodiment;

[0085] Figure 7 is a structural block diagram of an image processing device in one embodiment;

[0086] Figure 8 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0087] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0088] In one embodiment, Figure 1As shown, an image model processing method is provided. This embodiment uses the method applied to an electronic device as an example. The electronic device can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0089] In this embodiment, the image model processing method includes the following steps:

[0090] Step S102: semantically segment the first resolution image using a target semantic segmentation model, and obtain first intermediate semantic segmentation features of at least two different scales generated during the semantic segmentation process.

[0091] The target semantic segmentation model is a trained semantic segmentation model used to perform semantic segmentation on an image. The first resolution image is an image with a resolution lower than a preset resolution threshold, that is, a low-resolution image. The preset resolution threshold can be set as needed.

[0092] It can be understood that the target semantic segmentation model includes at least two convolutional layers, and semantic segmentation of the first resolution image can be performed through at least two convolutional layers to generate at least two intermediate semantic segmentation features of different scales. The first intermediate semantic segmentation feature reflects the semantic segmentation feature information of the first resolution image at the corresponding convolutional layer position, and the first intermediate semantic segmentation feature can expand the receptive field and improve the segmentation accuracy by capturing image spatial context information of different scales.

[0093] Optionally, the electronic device obtains a trained target semantic segmentation model, inputs the first resolution image into the target semantic segmentation model, performs semantic segmentation on the first resolution image through the target semantic segmentation model, and obtains first intermediate semantic segmentation features of at least two different scales generated during the semantic segmentation process.

[0094] Optionally, the electronic device constructs a super-resolution dataset and obtains a first resolution image from the super-resolution dataset. The electronic device selects high-quality reference images (Ground Truth, GT) from multiple mainstream super-resolution datasets, including but not limited to DIV2K and Flickr. The DIV2K dataset is a classic dataset in the field of super-resolution, containing high-resolution, high-quality images, and is widely used for training and evaluating super-resolution models. The Flickr dataset provides a large number of high-quality images taken in real scenes, which have rich details and textures and are very suitable as reference standards for super-resolution tasks. In addition to selecting high-quality reference images from existing datasets, a batch of high-quality and large reference images can be collected through real objects and scenes in the mobile phone shooting environment. These scenes include urban street scenes, natural scenery, indoor environments, etc., covering common shooting objects and scenes for mobile phone users. The electronic device performs degradation learning processing on the reference images in the super-resolution dataset to obtain the first resolution image corresponding to the reference image. Optionally, the electronic device can use advanced RealESRGAN, BSRGAN and unpaired dataset degradation learning processing to enhance the diversity and authenticity of the dataset.

[0095] Step S104, converting each first intermediate semantic segmentation feature into a first semantic relevance weight matrix; the elements in the first semantic relevance weight matrix represent the relevance between the corresponding position of the element in the first intermediate semantic segmentation feature and other positions.

[0096] Optionally, the elements in the first semantic relevance weight matrix represent the relevance between the corresponding position of the element and the adjacent position in the first intermediate semantic segmentation feature.

[0097] Optionally, the electronic device determines the correlation between each position and other positions in the first intermediate semantic segmentation feature; takes the correlation corresponding to each position as an element and generates a first semantic correlation weight matrix.

[0098] It can be understood that the higher the correlation between each position in the first intermediate semantic segmentation feature and other positions, the closer the semantic categories between the position and the other positions are, and they may be the same semantic category, while the lower the correlation between each position in the first intermediate semantic segmentation feature and other positions, the different semantic categories between the position and the other positions.

[0099] Optionally, each first intermediate semantic segmentation feature is converted into a first semantic relevance weight matrix, including: for each first intermediate semantic segmentation feature, convolving and upsampling the first intermediate semantic segmentation feature to obtain a first intermediate semantic segmentation feature of a target size; transposing the first intermediate semantic segmentation feature of the target size in terms of spatial dimension to obtain a transposed first intermediate semantic segmentation feature; and obtaining a first semantic relevance weight matrix based on the first intermediate semantic segmentation feature of the target size and the transposed first intermediate semantic segmentation feature.

[0100] Optionally, for each first intermediate semantic segmentation feature, the electronic device performs a first-path convolution and upsampling on the first intermediate semantic segmentation feature to obtain a first intermediate semantic segmentation feature of a target size, and performs a spatial dimension transposition processing on the first intermediate semantic segmentation feature of the target size to obtain a transposed first intermediate semantic segmentation feature; performs a second-path convolution and upsampling on the first intermediate semantic segmentation feature to obtain a first intermediate semantic segmentation feature of the target size; and obtains a first semantic correlation weight matrix based on the first intermediate semantic segmentation feature of the target size obtained in the second path and the transposed first intermediate semantic segmentation feature obtained in the first path.

[0101] Optionally, the electronic device performs matrix multiplication on the first intermediate semantic segmentation feature of the target size and the first intermediate semantic segmentation feature after transposition processing to obtain a first semantic relevance weight matrix.

[0102] Step S106, injecting each first semantic relevance weight matrix into the first super-resolution model, and super-resolution processing is performed on the first resolution image based on each first semantic relevance weight matrix through the first super-resolution model to obtain a super-resolution image; the resolution of the super-resolution image is greater than the resolution of the first resolution image.

[0103] Among them, the first super-resolution model is an untrained super-resolution model, which is used to super-resolution the image. The first super-resolution model can be trained using the RealESRGAN framework, and the generator can use classic super-resolution network structures such as RRDBNet and RCAN for experiments and comparisons. The generator is used to generate super-resolution images. The electronic device injects each first semantic relevance weight matrix into the first super-resolution model, and the first super-resolution model can make full use of semantic prior information to drive the super-resolution algorithm to reconstruct a more natural and realistic high-resolution image.

[0104] Optionally, the electronic device injects each first semantic relevance weight matrix into a convolutional layer of a corresponding scale of the first super-resolution model, and performs super-resolution processing on the first resolution image based on the first semantic relevance weight matrix through the convolutional layers of each scale in the first super-resolution model to obtain a super-resolution image.

[0105] Step S108, obtaining a loss based on the super-resolution image and the reference image corresponding to the first resolution image, and adjusting the first super-resolution model based on the loss to obtain a trained target super-resolution model.

[0106] The reference image is a true value image corresponding to the first resolution image, which is used to compare with the super-resolution image to determine the super-resolution processing effect of the first super-resolution model. The resolution of the reference image is greater than the resolution of the first resolution image.

[0107] Optionally, the electronic device adjusts the parameters of the first super-resolution model according to the loss, and returns to the step of performing semantic segmentation on the first resolution image through the target semantic segmentation model until the training cutoff condition is met, thereby obtaining a trained target super-resolution model. The training cutoff condition can be set as needed, such as when the loss is lower than a preset loss threshold, such as the number of training times or the training duration, etc., but is not limited thereto.

[0108] Optionally, the electronic device compares the super-resolved image with a reference image corresponding to the first resolution image, determines the similarity between the super-resolved image and the reference image, and determines the loss corresponding to the super-resolved image based on the similarity; the similarity and the loss are negatively correlated.

[0109] It can be understood that the higher the similarity between the super-resolution image and the reference image, the better the effect of the super-resolution model on the first resolution image, and the closer the super-resolution image is to the reference image, the smaller the loss.

[0110] Optionally, the electronic device obtains a super-resolution model driven by semantic priors based on the target semantic segmentation model and the target super-resolution model.

[0111] The electronic device combines the target semantic segmentation model and the target super-resolution model to obtain a super-resolution model driven by semantic priors.

[0112] Optionally, the second resolution image is super-resolved by a super-resolution model driven by semantic priors to obtain a target image; the super-resolution model driven by semantic priors includes a target semantic segmentation model and a target super-resolution model.

[0113] The electronic device obtains a super-resolution test set in a super-resolution data set, and uses the super-resolution test set to test a super-resolution model driven by semantic priors. If the test passes, the second resolution image is super-resolved by the super-resolution model driven by semantic priors to obtain a target image.

[0114] The electronic device inputs a low-resolution image in a super-resolution test set into a super-resolution model driven by semantic priors, and obtains a high-resolution image output by the super-resolution model driven by semantic priors; a test result is determined based on the high-resolution image and a reference image corresponding to the low-resolution image; if the test is determined to be passed based on the test result, the super-resolution model driven by semantic priors has high performance and effect, and can ensure that the resolution and detail fidelity of the image can be significantly improved in different scenarios.

[0115] The second resolution image is semantically segmented through the target semantic segmentation model, and at least two second intermediate semantic segmentation features of different scales generated in the semantic segmentation process are obtained; each second intermediate semantic segmentation feature is converted into a second semantic relevance weight matrix; the elements in the second semantic relevance weight matrix represent the correlation between the corresponding position of the element in the second intermediate semantic segmentation feature and other positions; each second semantic relevance weight matrix is ​​injected into the target super-resolution model, and the second resolution image is super-resolved based on each second semantic relevance weight matrix through the target super-resolution model to obtain a target image; the resolution of the target image is greater than the resolution of the second resolution image.

[0116] The above-mentioned image model processing method performs semantic segmentation on the first resolution image through the target semantic segmentation model, and obtains at least two first intermediate semantic segmentation features of different scales generated in the semantic segmentation process, and converts each first intermediate semantic segmentation feature into a first semantic relevance weight matrix. The elements in the first semantic relevance weight matrix represent the correlation between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature. Then, each first semantic relevance weight matrix is ​​injected into the first super-resolution model. The first super-resolution model can learn the correlation of the semantic positions in the first semantic relevance weight matrix, and perform feature consistency aggregation on pixels with consistent correlation, thereby ensuring that the texture distribution of the corresponding positions belongs to the same semantic category, and more accurately super-resolution processing is performed on the first resolution image, improving the details and consistency of the super-resolution processing in the super-resolution image, and better maintaining the semantic information and visual quality of the image, and then obtaining the loss according to the super-resolution image and the reference image corresponding to the first resolution image, and adjusting the first super-resolution model according to the loss to obtain a target super-resolution model with more accurate super-resolution processing.

[0117] By introducing the semantic prior information of the first intermediate semantic segmentation feature, the details in the image can be better preserved and enhanced, making the obtained super-resolution image closer to the real reference image in terms of visual effects and details. Different scenes (such as buildings, vegetation, etc.) have different requirements for super-resolution models. It is difficult for a single model to work well in multiple scenes. The semantic prior information of the first intermediate semantic segmentation feature can help the super-resolution model adjust the super-resolution strategy according to the actual scene and improve the generalization ability.

[0118] In one embodiment, a first resolution image is super-resolved based on each first semantic relevance weight matrix through a first super-resolution model to obtain a super-resolved image, including: super-resolving the first resolution image through the first super-resolution model to obtain a first intermediate super-resolution feature; obtaining a first semantic relevance weight matrix of a scale corresponding to the first intermediate super-resolution feature; obtaining a second intermediate super-resolution feature based on the first intermediate super-resolution feature and the corresponding first semantic relevance weight matrix; using the second intermediate super-resolution feature as a new first intermediate super-resolution feature, and returning to execute the step of obtaining the first semantic relevance weight matrix of a scale corresponding to the first intermediate super-resolution feature until a super-resolution image is obtained.

[0119] Among them, the first intermediate super-resolution feature and the second intermediate super-resolution feature are both feature maps.

[0120] Optionally, the electronic device obtains a second intermediate super-resolution feature output by the last convolutional layer in the first super-resolution model, and uses the second intermediate super-resolution feature output by the last convolutional layer as the super-resolution image.

[0121] Optionally, based on the first intermediate super-resolution feature and the corresponding first semantic relevance weight matrix, a second intermediate super-resolution feature is obtained, including: convolving the first intermediate super-resolution feature to obtain the first intermediate super-resolution feature after the convolution; performing matrix multiplication on the first intermediate super-resolution feature after the convolution and the corresponding first semantic relevance weight matrix to obtain the intermediate super-resolution feature of the second scale.

[0122] In this embodiment, the first resolution image is super-resolutioned by a first super-resolution model to obtain a first intermediate super-resolution feature, and a first semantic relevance weight matrix of a scale corresponding to the first intermediate super-resolution feature is obtained. Based on the first intermediate super-resolution feature and the corresponding first semantic relevance weight matrix, the correlation of semantic positions in the first semantic relevance weight matrix can be learned, and pixels with consistent correlations can be aggregated for feature consistency, thereby ensuring that the texture distribution at the corresponding positions belongs to the same semantic category, and the second intermediate super-resolution feature is obtained more accurately. This process is repeated until a super-resolution image with more accurate super-resolution is obtained.

[0123] Furthermore, the first intermediate super-resolution feature is convolved to obtain the first intermediate super-resolution feature after the convolution, and the first intermediate super-resolution feature after the convolution is matrix multiplied with the corresponding first semantic relevance weight matrix, so that the intermediate super-resolution feature of the second scale can be obtained more accurately.

[0124] In one embodiment, Figure 2As shown, for the first intermediate semantic segmentation feature A generated during the semantic segmentation process of the first resolution image by the target semantic segmentation model, the electronic device convolves and upsamples the first intermediate semantic segmentation feature A to obtain a first intermediate semantic segmentation feature of C×H×W three-dimensional size, and transposes the spatial dimension of the first intermediate semantic segmentation feature of C×H×W three-dimensional size to obtain the first intermediate semantic segmentation feature A' after the transposition processing; the first intermediate semantic segmentation feature of C×H×W three-dimensional size and the first intermediate semantic segmentation feature after the transposition processing are matrix multiplied to obtain a first semantic relevance weight matrix D, and the size of the first semantic relevance weight matrix D is (H×W)×(H×W); the electronic device obtains a first intermediate super-resolution feature B, and the first intermediate super-resolution feature B has the same scale as the first intermediate semantic segmentation feature A, convolves the first intermediate super-resolution feature B to obtain a first intermediate super-resolution feature after convolution, and matrix multiplies the first intermediate super-resolution feature after convolution with the corresponding first semantic relevance weight matrix D to obtain an intermediate super-resolution feature E of the second scale, and repeats this process in a loop until a super-resolution image is obtained. Among them, the first intermediate semantic segmentation feature is usually transformed into multiple scales through downsampling operation, that is, the spatial dimension size will change, so convolution and upsampling are required; while the spatial dimension of the first intermediate super-resolution feature is usually unchanged, so convolution is performed.

[0125] In one embodiment, the method further includes: obtaining a semantic segmentation result of a first resolution image through a target semantic segmentation model; obtaining a loss based on the super-resolution image and a reference image corresponding to the first resolution image, and adjusting the first super-resolution model based on the loss to obtain a trained target super-resolution model, including: dividing the super-resolution image into super-resolution image blocks of at least two semantic categories based on the semantic segmentation result; obtaining a loss based on the super-resolution image blocks of at least two semantic categories and a reference image corresponding to the first resolution image, and adjusting the first super-resolution model based on the loss to obtain a trained target super-resolution model.

[0126] Optionally, the electronic device obtains a first intermediate semantic segmentation feature output by the last convolution layer of the target semantic segmentation model, and uses the first intermediate semantic segmentation feature output by the last convolution layer as a semantic segmentation result of the first resolution image.

[0127] Optionally, the electronic device semantically divides the super-resolved image according to the semantic segmentation result to obtain irregular super-resolved image blocks of at least two semantic categories; each super-resolved image block includes a region of a semantic category in the super-resolved image.

[0128] Optionally, a loss is obtained based on super-resolution image blocks of at least two semantic categories and a reference image corresponding to the first resolution image, and the first super-resolution model is adjusted based on the loss to obtain a trained target super-resolution model, including: for each super-resolution image block of a semantic category, based on the super-resolution image block and the reference image corresponding to the first resolution image, determining a sub-discrimination result corresponding to the super-resolution image block; determining a discrimination result of the super-resolution image based on the sub-discrimination results corresponding to the super-resolution image blocks of each semantic category; obtaining a loss based on the discrimination result, and adjusting the first super-resolution model based on the loss to obtain a trained target super-resolution model.

[0129] For each super-resolved image block of a semantic category, the electronic device determines a reference image block in a reference image corresponding to the super-resolved image block, and determines a sub-discrimination result corresponding to the super-resolved image block based on the super-resolved image block and the reference image block.

[0130] Each sub-discrimination result includes a sub-loss corresponding to the super-resolved image block. The electronic device determines a discrimination result of the super-resolved image based on the sub-loss of each sub-discrimination result, and the discrimination result includes the loss of the super-resolved image.

[0131] Optionally, the electronic device may average at least two sub-losses to determine the loss of the super-resolution image; or may weighted average at least two sub-losses to determine the loss of the super-resolution image; or may use other methods to determine the loss of the super-resolution image, which are not limited here.

[0132] Optionally, if the loss is greater than a preset loss threshold, the parameters of the first super-resolution model are adjusted, and the step of performing semantic segmentation on the first resolution image through the target semantic segmentation model is returned to execute until the training cutoff condition is met, and a trained target super-resolution model is obtained; if the loss is less than or equal to the preset loss threshold, a trained target super-resolution model is obtained.

[0133] like Figure 3 As shown, the electronic device divides the super-resolved image into super-resolved image blocks of three semantic categories based on the semantic segmentation result; the super-resolved image blocks of the three semantic categories are input into a discriminator with shared parameters, and the discriminator determines the sub-discrimination results of the super-resolved image blocks of each semantic category respectively, and then determines the discrimination result of the super-resolved image according to the sub-discrimination results corresponding to the super-resolved image blocks of each semantic category.

[0134] In this embodiment, the electronic device obtains the semantic segmentation result of the first resolution image through the target semantic segmentation model; the explicit semantic segmentation result is introduced into the first super-resolution model, and the super-resolution image can be divided into super-resolution image blocks of at least two semantic categories based on the semantic segmentation result, so that the discriminator focuses on the judgment of a single semantic category, which not only improves the fine-grained discrimination ability of the discriminator, but also makes the discrimination more accurate, helps to reduce discrimination errors, and effectively improves the ability to judge the naturalness and authenticity of the super-resolution results, so that the output super-resolution image is more delicate and realistic visually, and provides a target super-resolution model with more accurate super-resolution processing after training.

[0135] In one embodiment, the above method also includes: obtaining a semantic segmentation dataset; using the semantic segmentation dataset to train an initial semantic segmentation model to obtain a trained first candidate semantic segmentation model; obtaining at least one second candidate semantic segmentation model different from the first candidate semantic segmentation model; and determining a target semantic segmentation model from the first candidate semantic segmentation model and at least one second candidate semantic segmentation model.

[0136] Among them, the semantic segmentation dataset is a dataset used to train the initial semantic segmentation model. The semantic segmentation dataset includes images with a resolution lower than a preset resolution threshold, that is, the semantic segmentation dataset includes low-resolution images, which are used to train the semantic segmentation model to perform semantic segmentation on low-resolution images. The semantic segmentation dataset includes datasets of at least two scenes. The semantic segmentation dataset can include multiple environments such as outdoor, indoor, and night scenes. At least two scenes can include city streets, parks, shopping malls, offices, home environments, night street scenes, etc., thereby ensuring that the semantic segmentation dataset has good representativeness and diversity.

[0137] The electronic device annotates the semantic segmentation dataset and divides the semantic categories of the semantic segmentation dataset into 8 main categories according to the texture characteristics of the images in the semantic segmentation dataset, namely: sky, building, vegetation, water surface, animal, night scene, face and other categories. The division of these categories not only takes into account the typical representatives of scene elements, but also fully considers the common objects and scenes in the images taken by electronic devices.

[0138] It is understandable that in specific application scenarios, especially for accurate semantic segmentation of low-resolution images taken with mobile phones, due to the poor imaging quality, shooting angle, ambient lighting conditions, etc. of mobile phone cameras, there is a significant domain distribution deviation between the constructed semantic segmentation datasets such as MS COCO, Pascal VOC, Cityscapes, etc. and the needs of actual applications. Therefore, a semantic segmentation dataset containing low-resolution images of multiple different scenes is constructed to train the semantic segmentation model to perform semantic segmentation on low-resolution images of different scenes.

[0139] Optionally, the semantic segmentation dataset includes a semantic segmentation training set, a semantic segmentation validation set, and a semantic segmentation test set; the semantic segmentation training set is used to train the initial semantic segmentation model, the semantic segmentation validation set is used to adjust parameters and evaluate the real-time performance of the model, and the semantic segmentation test set is used for the final performance evaluation to ensure the generalization ability of the model.

[0140] Optionally, the electronic device may divide the semantic segmentation dataset according to the following ratios: 70% of the data as a semantic segmentation training set, 15% of the data as a semantic segmentation validation set, and 15% of the data as a semantic segmentation test set.

[0141] The electronic device uses a semantic segmentation training set in a semantic segmentation dataset to perform semantic segmentation on an initial semantic segmentation model to obtain a semantic segmentation result; based on the semantic segmentation result and a semantic segmentation verification set, the initial semantic segmentation model is adjusted until a first candidate semantic segmentation model is obtained after training.

[0142] Optionally, the electronic device may train the initial semantic segmentation model by selecting a loss function or setting an optimizer, but is not limited thereto.

[0143] The second candidate semantic segmentation model may be an excellent model in the field of semantic segmentation, such as FCN, DeepLabv3+, PSPNet or HRNet, but is not limited thereto.

[0144] Optionally, the electronic device uses a semantic segmentation test set in the semantic segmentation dataset to test the first candidate semantic segmentation model and at least one second candidate semantic segmentation model to determine the test result of the first candidate semantic segmentation model and the test result of each second candidate semantic segmentation model; and determines the target semantic segmentation model based on the test results.

[0145] The test results of each model may include the Mean Intersection over Union (MIoU) indicator, based on which the model with the best test results may be determined as the target semantic segmentation model.

[0146] Optionally, the electronic device determines the model with the best semantic segmentation effect from the first candidate semantic segmentation model and at least one second candidate semantic segmentation model as the target semantic segmentation model. Optionally, the electronic device randomly determines a seat target semantic segmentation model from the first candidate semantic segmentation model and at least one second candidate semantic segmentation model. Optionally, the electronic device determines the model with the fastest semantic segmentation efficiency from the first candidate semantic segmentation model and at least one second candidate semantic segmentation model as the target semantic segmentation model. The electronic device can also determine the target semantic segmentation model in other ways, which are not limited here.

[0147] In this embodiment, the electronic device uses a semantic segmentation data set to train an initial semantic segmentation model to obtain a trained first candidate semantic segmentation model, and obtains at least one other second candidate semantic segmentation model, and then determines a target semantic segmentation model therefrom, thereby ensuring that the target semantic segmentation model is not only highly adaptable, but also can provide accurate and effective semantic information support for super-resolution tasks, thereby significantly improving the clarity and detail of the image.

[0148] In one embodiment, Figure 4 As shown, the electronic device inputs the first resolution image into the target semantic segmentation model and the first super-resolution model respectively, performs semantic segmentation on the first resolution image through the target semantic segmentation model, and obtains at least two first intermediate semantic segmentation features of different scales generated in the semantic segmentation process, converts each first intermediate semantic segmentation feature into a first semantic relevance weight matrix through an intermediate multi-scale segmentation feature injection unit, and injects each first semantic relevance weight matrix into the first super-resolution model, performs super-resolution processing on the first resolution image based on each first semantic relevance weight matrix through the first super-resolution model, and outputs a super-resolution image; outputs the semantic segmentation result of the first resolution image through the target semantic segmentation model, divides the super-resolution image into super-resolution image blocks of at least two semantic categories based on the semantic segmentation result, performs LOSS constraints according to the super-resolution image blocks of at least two semantic categories and the reference image corresponding to the first resolution image, that is, determines the loss, and adjusts the first super-resolution model according to the loss to obtain a trained target super-resolution model.

[0149] In one embodiment, another image model processing method is provided, which is applied to an electronic device and includes the following steps:

[0150] Step A1, obtain a semantic segmentation dataset; use the semantic segmentation dataset to train an initial semantic segmentation model to obtain a trained first candidate semantic segmentation model; obtain at least one second candidate semantic segmentation model different from the first candidate semantic segmentation model; determine a target semantic segmentation model from the first candidate semantic segmentation model and at least one second candidate semantic segmentation model.

[0151] Step A2: semantically segment the first resolution image using a target semantic segmentation model, and obtain first intermediate semantic segmentation features of at least two different scales generated during the semantic segmentation process.

[0152] Step A3: for each first intermediate semantic segmentation feature, convolve and upsample the first intermediate semantic segmentation feature to obtain the first intermediate semantic segmentation feature of the target size; transpose the first intermediate semantic segmentation feature of the target size in spatial dimension to obtain the transposed first intermediate semantic segmentation feature; based on the first intermediate semantic segmentation feature of the target size and the transposed first intermediate semantic segmentation feature, obtain the first semantic correlation weight matrix; the elements in the first semantic correlation weight matrix represent the correlation between the corresponding positions of the elements in the first intermediate semantic segmentation feature and other positions.

[0153] Step A4, injecting each first semantic relevance weight matrix into the first super-resolution model, and super-resolution the first resolution image through the first super-resolution model to obtain a first intermediate super-resolution feature; obtaining the first semantic relevance weight matrix of the scale corresponding to the first intermediate super-resolution feature; convolving the first intermediate super-resolution feature to obtain the first intermediate super-resolution feature after the convolution; performing matrix multiplication on the first intermediate super-resolution feature after the convolution and the corresponding first semantic relevance weight matrix to obtain the intermediate super-resolution feature of the second scale; using the second intermediate super-resolution feature as the new first intermediate super-resolution feature, and returning to execute the step of obtaining the first semantic relevance weight matrix of the scale corresponding to the first intermediate super-resolution feature until a super-resolution image is obtained.

[0154] Step A5, obtaining the semantic segmentation result of the first resolution image through the target semantic segmentation model; dividing the super-resolution image into super-resolution image blocks of at least two semantic categories based on the semantic segmentation result; obtaining the loss according to the super-resolution image blocks of at least two semantic categories and the reference image corresponding to the first resolution image, and adjusting the first super-resolution model according to the loss to obtain the trained target super-resolution model.

[0155] In one embodiment, Figure 5 As shown, an image processing method is provided. This embodiment uses the method applied to an electronic device as an example. The electronic device can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0156] In this embodiment, the image processing method includes the following steps:

[0157] Step S502: semantically segment the second resolution image using a target semantic segmentation model, and obtain at least two second intermediate semantic segmentation features of different scales generated during the semantic segmentation process.

[0158] The second resolution image is an image with a resolution lower than a preset resolution threshold, that is, a low-resolution image. The preset resolution threshold can be set as needed. The second intermediate semantic segmentation feature reflects the semantic segmentation feature information of the second resolution image at the corresponding convolutional layer position.

[0159] Optionally, the electronic device obtains a trained target semantic segmentation model, inputs the second resolution image into the target semantic segmentation model, performs semantic segmentation on the second resolution image through the target semantic segmentation model, and obtains at least two second intermediate semantic segmentation features of different scales generated during the semantic segmentation process.

[0160] Step S504, converting each second intermediate semantic segmentation feature into a second semantic relevance weight matrix; the elements in the second semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the second intermediate semantic segmentation feature.

[0161] Optionally, the elements in the second semantic relevance weight matrix represent the relevance between the corresponding position of the element and the adjacent position in the second intermediate semantic segmentation feature.

[0162] Optionally, the electronic device determines the correlation between each position and other positions in the second intermediate semantic segmentation feature; takes the correlation corresponding to each position as an element and generates a second semantic correlation weight matrix.

[0163] It can be understood that the higher the correlation between each position and other positions in the second intermediate semantic segmentation feature, the closer the semantic categories between the position and the other positions are, and they may be the same semantic category, while the lower the correlation between each position and other positions in the second intermediate semantic segmentation feature, the different semantic categories between the position and the other positions.

[0164] Optionally, each second intermediate semantic segmentation feature is converted into a second semantic relevance weight matrix, including: for each second intermediate semantic segmentation feature, convolving and upsampling the second intermediate semantic segmentation feature to obtain a second intermediate semantic segmentation feature of the target size; transposing the second intermediate semantic segmentation feature of the target size in terms of spatial dimension to obtain a transposed second intermediate semantic segmentation feature; and obtaining a second semantic relevance weight matrix based on the second intermediate semantic segmentation feature of the target size and the transposed second intermediate semantic segmentation feature.

[0165] Optionally, for each second intermediate semantic segmentation feature, the electronic device performs a first-path convolution and upsampling on the second intermediate semantic segmentation feature to obtain a second intermediate semantic segmentation feature of a target size, and performs a spatial dimension transposition processing on the second intermediate semantic segmentation feature of the target size to obtain a transposed second intermediate semantic segmentation feature; performs a second-path convolution and upsampling on the second intermediate semantic segmentation feature to obtain a second intermediate semantic segmentation feature of the target size; and obtains a second semantic correlation weight matrix based on the second intermediate semantic segmentation feature of the target size obtained in the second path and the transposed second intermediate semantic segmentation feature obtained in the first path.

[0166] Optionally, the electronic device performs matrix multiplication on the second intermediate semantic segmentation features of the target size and the second intermediate semantic segmentation features after transposition processing to obtain a second semantic relevance weight matrix.

[0167] Step S506, injecting each second semantic relevance weight matrix into the target super-resolution model, and super-resolution processing is performed on the second resolution image based on the each second semantic relevance weight matrix through the target super-resolution model to obtain a target image; the resolution of the target image is greater than the resolution of the second resolution image.

[0168] The target super-resolution model is a trained super-resolution model, which is used to super-resolution images to improve the resolution of images.

[0169] Optionally, the electronic device injects each second semantic relevance weight matrix into a convolutional layer of a corresponding scale of the second super-resolution model, and performs super-resolution processing on the second resolution image based on the second semantic relevance weight matrix through the convolutional layers of each scale in the second super-resolution model to obtain a target image.

[0170] The above-mentioned image processing method performs semantic segmentation on the second resolution image through the target semantic segmentation model, and obtains at least two second intermediate semantic segmentation features of different scales generated in the semantic segmentation process, and converts each second intermediate semantic segmentation feature into a second semantic relevance weight matrix. The elements in the second semantic relevance weight matrix represent the correlation between the corresponding position of the element in the second intermediate semantic segmentation feature and other positions. Then, each second semantic relevance weight matrix is ​​injected into the target super-resolution model. The target super-resolution model can learn the correlation of the semantic positions in the second semantic relevance weight matrix, and perform feature consistency aggregation on pixels with consistent correlation, thereby ensuring that the texture distribution of the corresponding positions belongs to the same semantic category, and more accurately super-resolution processing is performed on the second resolution image, thereby improving the details and consistency of the super-resolution processing in the image, better maintaining the semantic information and visual quality of the image, and obtaining a target super-resolution image with more accurate super-resolution processing.

[0171] In one embodiment, a target image is obtained by super-resolution processing the second resolution image based on each second semantic relevance weight matrix through a target super-resolution model, including: super-resolution processing the second resolution image through a second super-resolution model to obtain a second intermediate super-resolution feature; obtaining a second semantic relevance weight matrix of a scale corresponding to the second intermediate super-resolution feature; obtaining a second intermediate super-resolution feature based on the second intermediate super-resolution feature and the corresponding second semantic relevance weight matrix; using the second intermediate super-resolution feature as a new second intermediate super-resolution feature, and returning to execute the step of obtaining the second semantic relevance weight matrix of a scale corresponding to the second intermediate super-resolution feature until the target image is obtained.

[0172] Optionally, based on the second intermediate super-resolution features and the corresponding second semantic relevance weight matrix, a second intermediate super-resolution feature is obtained, including: convolving the second intermediate super-resolution feature to obtain the second intermediate super-resolution feature after the convolution; performing matrix multiplication on the second intermediate super-resolution feature after the convolution and the corresponding second semantic relevance weight matrix to obtain the intermediate super-resolution feature of the second scale.

[0173] In one embodiment, the above method also includes: obtaining a semantic segmentation dataset; using the semantic segmentation dataset to train an initial semantic segmentation model to obtain a trained first candidate semantic segmentation model; obtaining at least one second candidate semantic segmentation model different from the first candidate semantic segmentation model; and determining a target semantic segmentation model from the first candidate semantic segmentation model and at least one second candidate semantic segmentation model.

[0174] In one embodiment, obtaining a target super-resolution model includes: performing semantic segmentation on a first resolution image through a target semantic segmentation model, and obtaining at least two first intermediate semantic segmentation features of different scales generated during the semantic segmentation process; converting each first intermediate semantic segmentation feature into a first semantic relevance weight matrix; the elements in the first semantic relevance weight matrix represent the correlation between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature; injecting each first semantic relevance weight matrix into the first super-resolution model, and performing super-resolution processing on the first resolution image based on the first semantic relevance weight matrix through the first super-resolution model to obtain a super-resolution image; the resolution of the super-resolution image is greater than that of the first resolution image; obtaining a loss based on the super-resolution image and a reference image corresponding to the first resolution image, and adjusting the first super-resolution model based on the loss to obtain a trained target super-resolution model.

[0175] Optionally, super-resolution is performed on the first resolution image based on each first semantic relevance weight matrix through the first super-resolution model to obtain a super-resolution image, including: super-resolution is performed on the first resolution image through the first super-resolution model to obtain a first intermediate super-resolution feature; obtaining a first semantic relevance weight matrix of a scale corresponding to the first intermediate super-resolution feature; obtaining a second intermediate super-resolution feature based on the first intermediate super-resolution feature and the corresponding first semantic relevance weight matrix; using the second intermediate super-resolution feature as a new first intermediate super-resolution feature, and returning to execute the step of obtaining the first semantic relevance weight matrix of a scale corresponding to the first intermediate super-resolution feature until a super-resolution image is obtained.

[0176] Optionally, based on the first intermediate super-resolution feature and the corresponding first semantic relevance weight matrix, a second intermediate super-resolution feature is obtained, including: convolving the first intermediate super-resolution feature to obtain the first intermediate super-resolution feature after the convolution; performing matrix multiplication on the first intermediate super-resolution feature after the convolution and the corresponding first semantic relevance weight matrix to obtain the intermediate super-resolution feature of the second scale.

[0177] Optionally, each first intermediate semantic segmentation feature is converted into a first semantic relevance weight matrix, including: for each first intermediate semantic segmentation feature, convolving and upsampling the first intermediate semantic segmentation feature to obtain a first intermediate semantic segmentation feature of a target size; transposing the first intermediate semantic segmentation feature of the target size in terms of spatial dimension to obtain a transposed first intermediate semantic segmentation feature; and obtaining a first semantic relevance weight matrix based on the first intermediate semantic segmentation feature of the target size and the transposed first intermediate semantic segmentation feature.

[0178] Optionally, the above method also includes: obtaining a semantic segmentation result of the first resolution image through a target semantic segmentation model; obtaining a loss based on the super-resolution image and a reference image corresponding to the first resolution image, and adjusting the first super-resolution model based on the loss to obtain a trained target super-resolution model, including: dividing the super-resolution image into super-resolution image blocks of at least two semantic categories based on the semantic segmentation result; obtaining a loss based on the super-resolution image blocks of at least two semantic categories and a reference image corresponding to the first resolution image, and adjusting the first super-resolution model based on the loss to obtain a trained target super-resolution model.

[0179] Optionally, a loss is obtained based on super-resolution image blocks of at least two semantic categories and a reference image corresponding to the first resolution image, and the first super-resolution model is adjusted based on the loss to obtain a trained target super-resolution model, including: for each super-resolution image block of a semantic category, based on the super-resolution image block and the reference image corresponding to the first resolution image, determining a sub-discrimination result corresponding to the super-resolution image block; determining a discrimination result of the super-resolution image based on the sub-discrimination results corresponding to the super-resolution image blocks of each semantic category; obtaining a loss based on the discrimination result, and adjusting the first super-resolution model based on the loss to obtain a trained target super-resolution model.

[0180] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0181] Based on the same inventive concept, the embodiment of the present application also provides an image model processing device for implementing the above-mentioned image model processing method, and an image processing device for implementing the above-mentioned image processing method. The implementation solutions to the problems provided by the image model processing device and the image processing device are similar to the implementation solutions recorded in the above-mentioned method, so the specific limitations in one or more of the image model processing device embodiments provided below can refer to the limitations on the image model processing method above, and the specific limitations in one or more of the image processing device embodiments provided can refer to the limitations on the image processing method above, and will not be repeated here.

[0182] In an exemplary embodiment, Figure 6As shown, an image model processing device is provided, comprising: a first semantic segmentation module 602, a first semantic relevance weight matrix acquisition module 604, a first super-resolution processing module 606 and a target super-resolution model acquisition module 608, wherein:

[0183] The first semantic segmentation module 602 is used to perform semantic segmentation on the first resolution image through a target semantic segmentation model, and obtain at least two first intermediate semantic segmentation features of different scales generated during the semantic segmentation process.

[0184] The first semantic relevance weight matrix acquisition module 604 is used to convert each first intermediate semantic segmentation feature into a first semantic relevance weight matrix; the elements in the first semantic relevance weight matrix represent the correlation between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature.

[0185] The first super-resolution processing module 606 is used to inject each first semantic relevance weight matrix into the first super-resolution model, and perform super-resolution processing on the first resolution image based on the first semantic relevance weight matrix through the first super-resolution model to obtain a super-resolution image; the resolution of the super-resolution image is greater than the resolution of the first resolution image.

[0186] The target super-resolution model acquisition module 608 is used to obtain a loss based on the super-resolution image and the reference image corresponding to the first resolution image, and adjust the first super-resolution model based on the loss to obtain a trained target super-resolution model.

[0187] The above-mentioned image model processing device performs semantic segmentation on the first resolution image through the target semantic segmentation model, and obtains at least two first intermediate semantic segmentation features of different scales generated in the semantic segmentation process, and converts each first intermediate semantic segmentation feature into a first semantic relevance weight matrix, the elements in the first semantic relevance weight matrix represent the correlation between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature, then, each first semantic relevance weight matrix is ​​injected into the first super-resolution model, the first super-resolution model can learn the correlation of the semantic positions in the first semantic relevance weight matrix, and perform feature consistency aggregation on pixels with consistent correlation, so as to ensure that the texture distribution of the corresponding positions belongs to the same semantic category, and more accurately super-resolution the first resolution image, improve the details and consistency of the super-resolution processing in the super-resolution image, better maintain the semantic information and visual quality of the image, and then obtain the loss according to the super-resolution image and the reference image corresponding to the first resolution image, and adjust the first super-resolution model according to the loss to obtain a target super-resolution model with more accurate super-resolution processing.

[0188] In one embodiment, the first super-resolution processing module 606 is also used to perform super-resolution processing on the first resolution image through the first super-resolution model to obtain a first intermediate super-resolution feature; obtain a first semantic relevance weight matrix of a scale corresponding to the first intermediate super-resolution feature; obtain a second intermediate super-resolution feature based on the first intermediate super-resolution feature and the corresponding first semantic relevance weight matrix; use the second intermediate super-resolution feature as a new first intermediate super-resolution feature, and obtain the first semantic relevance weight matrix of a scale corresponding to the first intermediate super-resolution feature, until a super-resolution image is obtained.

[0189] In one embodiment, the first super-resolution processing module 606 is further used to convolve the first intermediate super-resolution feature to obtain a first intermediate super-resolution feature after convolution; and to perform matrix multiplication on the first intermediate super-resolution feature after convolution and the corresponding first semantic relevance weight matrix to obtain an intermediate super-resolution feature of a second scale.

[0190] In one embodiment, the first semantic relevance weight matrix acquisition module 604 is also used to convolve and upsample each first intermediate semantic segmentation feature to obtain a first intermediate semantic segmentation feature of a target size; transpose the first intermediate semantic segmentation feature of the target size in terms of spatial dimension to obtain a transposed first intermediate semantic segmentation feature; and obtain a first semantic relevance weight matrix based on the first intermediate semantic segmentation feature of the target size and the transposed first intermediate semantic segmentation feature.

[0191] In one embodiment, the above-mentioned device also includes a semantic segmentation result acquisition module, which is used to obtain the semantic segmentation result of the first resolution image through the target semantic segmentation model; the above-mentioned target super-resolution model acquisition module 608 is also used to divide the super-resolution image into super-resolution image blocks of at least two semantic categories based on the semantic segmentation result; obtain the loss according to the super-resolution image blocks of the at least two semantic categories and the reference image corresponding to the first resolution image, and adjust the first super-resolution model according to the loss to obtain the trained target super-resolution model.

[0192] In one embodiment, the target super-resolution model acquisition module 608 is also used to determine, for each semantic category, a sub-discrimination result corresponding to the super-resolution image block based on the super-resolution image block and a reference image corresponding to the first resolution image; determine the discrimination result of the super-resolution image according to the sub-discrimination results corresponding to the super-resolution image blocks of each semantic category; obtain a loss based on the discrimination result, and adjust the first super-resolution model according to the loss to obtain a trained target super-resolution model.

[0193] In one embodiment, the above-mentioned device also includes a target semantic segmentation model acquisition module, which is used to obtain a semantic segmentation dataset; use the semantic segmentation dataset to train the initial semantic segmentation model to obtain a trained first candidate semantic segmentation model; obtain at least one second candidate semantic segmentation model different from the first candidate semantic segmentation model; and determine the target semantic segmentation model from the first candidate semantic segmentation model and the at least one second candidate semantic segmentation model.

[0194] In an exemplary embodiment, Figure 7 As shown, an image processing device is provided, including: a second semantic segmentation module 702, a second semantic relevance weight matrix acquisition module 704, and a second super-resolution processing module 706, wherein:

[0195] The second semantic segmentation module 702 is used to perform semantic segmentation on the second resolution image through a target semantic segmentation model, and obtain at least two second intermediate semantic segmentation features of different scales generated during the semantic segmentation process.

[0196] The second semantic relevance weight matrix acquisition module 704 is used to convert each second intermediate semantic segmentation feature into a second semantic relevance weight matrix; the elements in the second semantic relevance weight matrix represent the correlation between the corresponding position of the element in the second intermediate semantic segmentation feature and other positions.

[0197] The second super-resolution processing module 706 is used to inject each second semantic relevance weight matrix into the target super-resolution model, and perform super-resolution processing on the second resolution image based on the second semantic relevance weight matrix through the target super-resolution model to obtain a target image; the resolution of the target image is greater than the resolution of the second resolution image.

[0198] The above-mentioned image processing device performs semantic segmentation on the second resolution image through the target semantic segmentation model, and obtains at least two second intermediate semantic segmentation features of different scales generated in the semantic segmentation process, and converts each second intermediate semantic segmentation feature into a second semantic relevance weight matrix, wherein the elements in the second semantic relevance weight matrix represent the correlation between the corresponding position of the element in the second intermediate semantic segmentation feature and other positions. Then, each second semantic relevance weight matrix is ​​injected into the target super-resolution model. The target super-resolution model can learn the correlation of the semantic positions in the second semantic relevance weight matrix, and perform feature consistency aggregation on pixels with consistent correlation, thereby ensuring that the texture distribution of the corresponding positions belongs to the same semantic category, and more accurately super-resolution processing is performed on the second resolution image, thereby improving the details and consistency of the super-resolution processing in the image, better maintaining the semantic information and visual quality of the image, and obtaining a target super-resolution image with more accurate super-resolution processing.

[0199] In one embodiment, the second super-resolution processing module 706 is also used to perform super-resolution processing on the second resolution image through a second super-resolution model to obtain a second intermediate super-resolution feature; obtain a second semantic relevance weight matrix of a scale corresponding to the second intermediate super-resolution feature; obtain the second intermediate super-resolution feature based on the second intermediate super-resolution feature and the corresponding second semantic relevance weight matrix; use the second intermediate super-resolution feature as a new second intermediate super-resolution feature, and return to execute the step of obtaining the second semantic relevance weight matrix of a scale corresponding to the second intermediate super-resolution feature until the target image is obtained.

[0200] In one embodiment, the second super-resolution processing module 706 is further used to convolve the second intermediate super-resolution features to obtain the second intermediate super-resolution features after the convolution; and perform matrix multiplication on the second intermediate super-resolution features after the convolution and the corresponding second semantic relevance weight matrix to obtain the intermediate super-resolution features of the second scale.

[0201] In one embodiment, the above-mentioned device also includes a target semantic segmentation model acquisition module, which is used to obtain a semantic segmentation dataset; use the semantic segmentation dataset to train the initial semantic segmentation model to obtain a trained first candidate semantic segmentation model; obtain at least one second candidate semantic segmentation model different from the first candidate semantic segmentation model; determine the target semantic segmentation model from the first candidate semantic segmentation model and at least one second candidate semantic segmentation model.

[0202] In one embodiment, the apparatus further comprises a first semantic segmentation module, a first semantic relevance weight matrix acquisition module, a first super-resolution processing module and a target super-resolution model acquisition module;

[0203] A first semantic segmentation module, configured to perform semantic segmentation on the first resolution image by using a target semantic segmentation model, and obtain first intermediate semantic segmentation features of at least two different scales generated during the semantic segmentation process;

[0204] The first semantic relevance weight matrix acquisition module is used to convert each first intermediate semantic segmentation feature into a first semantic relevance weight matrix; the elements in the first semantic relevance weight matrix represent the correlation between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature.

[0205] A first super-resolution processing module is used to inject each first semantic relevance weight matrix into a first super-resolution model, and perform super-resolution processing on the first resolution image based on each first semantic relevance weight matrix through the first super-resolution model to obtain a super-resolution image; the resolution of the super-resolution image is greater than the resolution of the first resolution image;

[0206] The target super-resolution model acquisition module is used to obtain the loss according to the super-resolution image and the reference image corresponding to the first resolution image, and adjust the first super-resolution model according to the loss to obtain the trained target super-resolution model.

[0207] In one embodiment, the above-mentioned first super-resolution processing module is also used to perform super-resolution processing on the first resolution image through the first super-resolution model to obtain a first intermediate super-resolution feature; obtain a first semantic relevance weight matrix of the scale corresponding to the first intermediate super-resolution feature; obtain a second intermediate super-resolution feature based on the first intermediate super-resolution feature and the corresponding first semantic relevance weight matrix; use the second intermediate super-resolution feature as a new first intermediate super-resolution feature, and obtain the first semantic relevance weight matrix of the scale corresponding to the first intermediate super-resolution feature, until a super-resolution image is obtained.

[0208] In one embodiment, the first super-resolution processing module is further used to convolve the first intermediate super-resolution feature to obtain a first intermediate super-resolution feature after convolution; and to perform matrix multiplication on the first intermediate super-resolution feature after convolution and the corresponding first semantic relevance weight matrix to obtain an intermediate super-resolution feature of a second scale.

[0209] In one embodiment, the above-mentioned first semantic relevance weight matrix acquisition module is also used to convolve and upsample the first intermediate semantic segmentation features for each first intermediate semantic segmentation feature to obtain the first intermediate semantic segmentation feature of the target size; transpose the first intermediate semantic segmentation feature of the target size in terms of spatial dimension to obtain the transposed first intermediate semantic segmentation feature; and obtain the first semantic relevance weight matrix based on the first intermediate semantic segmentation feature of the target size and the transposed first intermediate semantic segmentation feature.

[0210] In one embodiment, the above-mentioned device also includes a semantic segmentation result acquisition module, which is used to obtain the semantic segmentation result of the first resolution image through the target semantic segmentation model; the above-mentioned target super-resolution model acquisition module is also used to divide the super-resolution image into super-resolution image blocks of at least two semantic categories based on the semantic segmentation result; obtain the loss according to the super-resolution image blocks of the at least two semantic categories and the reference image corresponding to the first resolution image, and adjust the first super-resolution model according to the loss to obtain the trained target super-resolution model.

[0211] In one embodiment, the target super-resolution model acquisition module is also used to determine, for each semantic category of the super-resolution image block, a sub-discrimination result corresponding to the super-resolution image block based on the super-resolution image block and a reference image corresponding to the first resolution image; determine the discrimination result of the super-resolution image according to the sub-discrimination results corresponding to the super-resolution image blocks of each semantic category; obtain a loss based on the discrimination result, and adjust the first super-resolution model according to the loss to obtain a trained target super-resolution model.

[0212] The above-mentioned image model processing device or each module in the image processing device can be implemented in whole or in part by software, hardware or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the electronic device in the form of hardware, or can be stored in the memory in the electronic device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0213] In an exemplary embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, an image model processing or an image processing method is implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.

[0214] Those skilled in the art will understand that Figure 8The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0215] In one embodiment, an electronic device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0216] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0217] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0218] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0219] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0220] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0221] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image model processing method, characterized in that: The method comprises: Performing semantic segmentation on the first resolution image through a target semantic segmentation model, and obtaining first intermediate semantic segmentation features of at least two different scales generated in the semantic segmentation process; Converting each first intermediate semantic segmentation feature into a first semantic relevance weight matrix; the elements in the first semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature; Injecting each first semantic relevance weight matrix into a first super-resolution model, and performing super-resolution processing on the first resolution image based on each first semantic relevance weight matrix through the first super-resolution model to obtain a super-resolution image; the resolution of the super-resolution image is greater than the resolution of the first resolution image; A loss is obtained according to the super-resolution image and a reference image corresponding to the first resolution image, and the first super-resolution model is adjusted according to the loss to obtain a trained target super-resolution model.

2. The method according to claim 1, characterized in that The step of performing super-resolution processing on the first resolution image based on each of the first semantic relevance weight matrices by using the first super-resolution model to obtain a super-resolution image includes: Performing super-resolution processing on the first resolution image by using the first super-resolution model to obtain a first intermediate super-resolution feature; Obtain a first semantic relevance weight matrix of a scale corresponding to the first intermediate super-resolution feature; Obtaining a second intermediate super-resolution feature based on the first intermediate super-resolution feature and the corresponding first semantic relevance weight matrix; The second intermediate super-resolution feature is used as a new first intermediate super-resolution feature, and the step of obtaining a first semantic relevance weight matrix corresponding to a scale of the first intermediate super-resolution feature is returned to be executed until a super-resolution image is obtained.

3. The method according to claim 2, characterized in that The obtaining, based on the first intermediate super-resolution feature and the corresponding first semantic relevance weight matrix, a second intermediate super-resolution feature comprises: Convolving the first intermediate super-resolution feature to obtain a first intermediate super-resolution feature after convolution; The first intermediate super-resolution feature after the convolution is matrix-multiplied with the corresponding first semantic relevance weight matrix to obtain an intermediate super-resolution feature of a second scale.

4. The method according to claim 1, characterized in that: The step of converting each first intermediate semantic segmentation feature into a first semantic relevance weight matrix comprises: For each first intermediate semantic segmentation feature, convolve and upsample the first intermediate semantic segmentation feature to obtain a first intermediate semantic segmentation feature of a target size; Performing a transposition process on the first intermediate semantic segmentation feature of the target size in terms of spatial dimension to obtain a transposed first intermediate semantic segmentation feature; A first semantic relevance weight matrix is ​​obtained based on the first intermediate semantic segmentation features of the target size and the transposed first intermediate semantic segmentation features.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Acquire a semantic segmentation result of the first resolution image by using the target semantic segmentation model; The step of obtaining a loss according to the super-resolution image and a reference image corresponding to the first resolution image, and adjusting the first super-resolution model according to the loss to obtain a trained target super-resolution model includes: Dividing the super-resolved image into super-resolved image blocks of at least two semantic categories based on the semantic segmentation result; A loss is obtained according to the super-resolution image blocks of the at least two semantic categories and a reference image corresponding to the first resolution image, and the first super-resolution model is adjusted according to the loss to obtain a trained target super-resolution model.

6. The method according to claim 5, characterized in that The step of obtaining a loss according to the super-resolution image blocks of the at least two semantic categories and a reference image corresponding to the first resolution image, and adjusting the first super-resolution model according to the loss to obtain a trained target super-resolution model includes: For each super-resolved image block of each semantic category, based on the super-resolved image block and a reference image corresponding to the first resolution image, determining a sub-discrimination result corresponding to the super-resolved image block; Determining a discrimination result of the super-resolved image according to the sub-discrimination results corresponding to the super-resolved image blocks of each semantic category; A loss is obtained based on the discrimination result, and the first super-resolution model is adjusted according to the loss to obtain a trained target super-resolution model.

7. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Get the semantic segmentation dataset; Using the semantic segmentation dataset to train the initial semantic segmentation model to obtain a trained first candidate semantic segmentation model; Acquire at least one second candidate semantic segmentation model different from the first candidate semantic segmentation model; A target semantic segmentation model is determined from the first candidate semantic segmentation model and the at least one second candidate semantic segmentation model.

8. An image processing method, characterized in that: The method comprises: Performing semantic segmentation on the second resolution image through the target semantic segmentation model, and obtaining at least two second intermediate semantic segmentation features of different scales generated in the semantic segmentation process; Converting each second intermediate semantic segmentation feature into a second semantic relevance weight matrix; the elements in the second semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the second intermediate semantic segmentation feature; Each second semantic relevance weight matrix is ​​injected into a target super-resolution model, and the second resolution image is super-resolved based on each second semantic relevance weight matrix through the target super-resolution model to obtain a target image; the resolution of the target image is greater than the resolution of the second resolution image.

9. The method according to claim 8, characterized in that The target super-resolution model is obtained according to any one of the methods of claims 1 to 7.

10. An image model processing device, characterized in that: The device comprises: A first semantic segmentation module, configured to perform semantic segmentation on the first resolution image by using a target semantic segmentation model, and obtain first intermediate semantic segmentation features of at least two different scales generated during the semantic segmentation process; A first semantic relevance weight matrix acquisition module is used to convert each first intermediate semantic segmentation feature into a first semantic relevance weight matrix; the elements in the first semantic relevance weight matrix represent the relevance between the corresponding position of the element and other positions in the first intermediate semantic segmentation feature; A first super-resolution processing module is used to inject each first semantic relevance weight matrix into a first super-resolution model, and perform super-resolution processing on the first resolution image based on each first semantic relevance weight matrix through the first super-resolution model to obtain a super-resolution image; the resolution of the super-resolution image is greater than the resolution of the first resolution image; The target super-resolution model acquisition module is used to obtain the loss according to the super-resolution image and the reference image corresponding to the first resolution image, and adjust the first super-resolution model according to the loss to obtain the trained target super-resolution model.

11. An image processing device, characterized in that: The device comprises: A second semantic segmentation module is used to perform semantic segmentation on the second resolution image by using a target semantic segmentation model, and obtain at least two second intermediate semantic segmentation features of different scales generated in the semantic segmentation process; A second semantic relevance weight matrix acquisition module is used to convert each second intermediate semantic segmentation feature into a second semantic relevance weight matrix; the elements in the second semantic relevance weight matrix represent the correlation between the corresponding position of the element and other positions in the second intermediate semantic segmentation feature; The second super-resolution processing module is used to inject each second semantic relevance weight matrix into the target super-resolution model, and perform super-resolution processing on the second resolution image based on the second semantic relevance weight matrix through the target super-resolution model to obtain a target image; the resolution of the target image is greater than the resolution of the second resolution image.

12. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images