An image resolution adjustment method, device, equipment and storage medium

Abstract

The application discloses an image resolution adjusting method, device and equipment and a storage medium. The method comprises the following steps: obtaining an image to be adjusted and a first target resolution; the resolution of the image to be adjusted is higher than the first target resolution; dividing the image to be adjusted into N candidate images with the first target resolution; wherein, each pixel point information in the image to be adjusted is contained in any candidate image; N is a positive integer and N>1; extracting edge feature information for the obtained candidate images; and comprehensively obtaining the target image with the first target resolution by combining the N obtained candidate images and the extracted edge feature information.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image resolution adjustment method, apparatus, device, and storage medium. Background Technology

[0002] In practical image processing, it is often necessary to adjust the image resolution. For example, for display screens of different sizes, it is usually necessary to adjust the resolution of a fixed-size image to fit the display screen. Specifically, this can be done by lowering or raising the image resolution.

[0003] In related technologies, some pixels in an image can be directly deleted to reduce the image resolution, or some pixels in an image can be directly copied to increase the image resolution.

[0004] For example, when reducing image resolution, simply delete pixels in odd-numbered rows or columns; when increasing image resolution, simply copy pixels in odd-numbered rows or columns.

[0005] However, using these methods to adjust image resolution often results in poor display quality after adjustment.

[0006] For example, when reducing image resolution, directly deleting some pixels results in the loss of some information, leading to lower image clarity and poor display quality; when increasing image resolution, directly copying some pixels and adding them to the image causes image distortion and poor display quality. Summary of the Invention

[0007] This invention provides an image resolution adjustment method, apparatus, device, and storage medium to address the shortcomings of related technologies.

[0008] According to a first aspect of the present invention, an image resolution adjustment method is provided, comprising:

[0009] Obtain the image to be adjusted and a first target resolution; the resolution of the image to be adjusted is higher than the first target resolution;

[0010] The image to be adjusted is divided into N candidate images with a resolution equal to the first target resolution; wherein, the information of each pixel in the image to be adjusted is contained in any candidate image; N is a positive integer and N>1;

[0011] For the segmented candidate images, extract edge feature information;

[0012] By combining the N candidate images obtained from the segmentation with the extracted edge feature information, a target image with a resolution equal to the first target resolution is obtained.

[0013] Optionally, dividing the image to be adjusted into N candidate images with a resolution equal to the first target resolution includes:

[0014] Based on the first target resolution, several pixel matrices of the same size are divided from the image to be adjusted; each pixel matrix includes N pixels.

[0015] In the image to be adjusted, the number of single-row pixel matrices divided horizontally is the same as the number of single-row pixels of the first target resolution, and the number of single-column pixel matrices divided vertically is the same as the number of single-column pixels of the first target resolution.

[0016] Based on the divided pixel matrix, N candidate images with a resolution equal to the first target resolution are extracted.

[0017] Optionally, pixels at the same position in different candidate images belong to the same pixel matrix; different pixels in any candidate image are located at the same position in their respective pixel matrices.

[0018] Optionally, extracting edge feature information from the segmented candidate images includes:

[0019] Select any candidate image as the starting image;

[0020] For the initial image, using N-1 preset convolutional kernels corresponding to N-1 other candidate images, N-1 edge feature images are extracted for each of the N-1 other candidate images; the resolution of the edge feature images is the first target resolution;

[0021] Different candidate images correspond to different preset convolution kernels; the preset convolution kernels are used to extract gradient feature information between pixels in a preset direction; the preset direction is the relative direction of different pixels at the same position in the same pixel matrix between the corresponding candidate image and the starting image.

[0022] Optionally, the process of combining the N candidate images obtained from the comprehensive division with the extracted edge feature information to obtain a target image with a resolution equal to the first target resolution includes:

[0023] The N candidate images obtained from the segmentation and the N-1 edge feature images extracted are input into a pre-trained image channel merging network to obtain a target image with a resolution of the first target resolution output by the image channel merging network.

[0024] Optionally, the image channel merging network is used for:

[0025] For the N-1 other candidate images, each image is superimposed with its corresponding edge feature image to obtain N-1 superimposed images;

[0026] For the N-1 superimposed images and the starting image, features are extracted using the representation layer to obtain N convolutional feature images;

[0027] The N convolutional feature images are superimposed on each other by channels, and then the channel merging layer is used to extract features to obtain a target image with the same resolution as the first target resolution and the same number of channels as the image to be adjusted.

[0028] Optionally, extracting edge feature information from the segmented candidate images includes:

[0029] For each identical location among the N candidate images obtained from the segmentation, perform the following operations to obtain a max-pooling feature map with a resolution of the first target resolution, and determine the max-pooling feature map as the extracted edge feature information:

[0030] The pixel value with the largest value among the pixel values ​​of the N pixels at the target position in the N candidate images is determined as the pixel value of the pixel at the target position in the max pooling feature map.

[0031] Optionally, the process of combining the N candidate images obtained from the comprehensive division with the extracted edge feature information to obtain a target image with a resolution equal to the first target resolution includes:

[0032] The N candidate images obtained from the segmentation and the max pooling feature map are input into a pre-trained image channel fusion network to obtain a target image with a resolution of the first target resolution output by the image channel fusion network.

[0033] Optionally, the image channel fusion network is used for:

[0034] For the N candidate images, each image is superimposed with the max pooling feature map through channels to obtain N superimposed results;

[0035] For the N superimposed results, features are extracted using the representation layer to obtain N alternative images with the same resolution as the first target resolution and the same number of channels as the image to be adjusted.

[0036] The N candidate images are input into a max pooling layer to obtain a target image with the same resolution as the first target resolution and the same number of channels as the image to be adjusted.

[0037] Optionally, the max pooling layer is used for:

[0038] For each identical position among the N candidate images, perform the following operations to obtain a target image with the same resolution as the first target resolution and the same number of channels as the image to be adjusted:

[0039] The pixel value with the largest value among the pixel values ​​of the N pixels at the target position in the N candidate images is determined as the pixel value of the pixel at the target position in the target image.

[0040] According to a second aspect of the present invention, another image resolution adjustment method is provided, comprising:

[0041] Obtain the image to be adjusted and the second target resolution; the resolution of the image to be adjusted is lower than the resolution of the second target resolution;

[0042] Based on the second target resolution and the resolution of the image to be adjusted, determine M pixel expansion directions; M is a positive integer and M≥1; wherein, the product of (M+1) and the resolution of the image to be adjusted is greater than or equal to the second target resolution;

[0043] For the image to be adjusted, obtain M edge feature information in the M pixel expansion directions respectively;

[0044] By combining the image to be adjusted and the M edge feature information, a target image with the resolution of the second target is obtained.

[0045] Optionally, for the image to be adjusted, obtaining M edge feature information in the M pixel expansion directions respectively includes:

[0046] For the image to be adjusted, M edge feature images are extracted using M preset convolution kernels corresponding to the M pixel expansion directions; the resolution of the edge feature images is the same as the resolution of the image to be adjusted.

[0047] Different pixel expansion directions correspond to different preset convolution kernels; the preset convolution kernels are used to extract the gradient feature information between pixels in the corresponding pixel expansion direction.

[0048] Optionally, the step of integrating the image to be adjusted and the M edge feature information to obtain a target image with a resolution equal to the second target resolution includes:

[0049] The image to be adjusted and the M edge feature images are input into a pre-trained image combination network to obtain a target image with the resolution of the second target resolution output by the image combination network; the target image has the same number of channels as the image to be adjusted.

[0050] Optionally, the image combining network is used for:

[0051] For the M edge feature images, each is superimposed with the image to be adjusted through channels to obtain M superimposed results;

[0052] For the M superposition results and the image to be adjusted, features are extracted using the representation layer to obtain M+1 images to be combined with the same resolution and number of channels as the image to be adjusted;

[0053] The M+1 images to be combined are input into the combination layer to obtain a target image with the second target resolution and the same number of channels as the images to be adjusted.

[0054] Optionally, the combined layer is used for:

[0055] Perform the following operations on each pixel in the image to be combined corresponding to the image to be adjusted, to obtain the combined result:

[0056] Starting from the target pixel, a pixel matrix containing M+1 pixels is obtained by expanding based on the M pixel expansion directions, and the pixels in the other M images to be combined that are at the same position as the target pixel are added to the expanded pixel matrix.

[0057] Based on the obtained combination results, a target image with the second target resolution and the same number of channels as the image to be adjusted is obtained through preset image processing.

[0058] According to a third aspect of the present invention, an image resolution adjustment device is provided, comprising:

[0059] The first acquisition unit is used to acquire the image to be adjusted and the first target resolution; the resolution of the image to be adjusted is higher than the first target resolution.

[0060] A segmentation unit is used to divide the image to be adjusted into N candidate images with a resolution of the first target resolution; wherein, the information of each pixel in the image to be adjusted is contained in any candidate image; N is a positive integer and N>1;

[0061] The first feature unit is used to extract edge feature information from the segmented candidate image;

[0062] The first integration unit is used to integrate the N candidate images obtained from the division and the extracted edge feature information to obtain a target image with a resolution of the first target resolution.

[0063] Optionally, the partitioning unit is used for:

[0064] Based on the first target resolution, several pixel matrices of the same size are divided from the image to be adjusted; each pixel matrix includes N pixels.

[0065] In the image to be adjusted, the number of single-row pixel matrices divided horizontally is the same as the number of single-row pixels of the first target resolution, and the number of single-column pixel matrices divided vertically is the same as the number of single-column pixels of the first target resolution.

[0066] Based on the divided pixel matrix, N candidate images with a resolution equal to the first target resolution are extracted.

[0067] Optionally, pixels at the same position in different candidate images belong to the same pixel matrix; different pixels in any candidate image are located at the same position in their respective pixel matrices.

[0068] Optionally, the first feature unit is used for:

[0069] Select any candidate image as the starting image;

[0070] For the initial image, using N-1 preset convolutional kernels corresponding to N-1 other candidate images, N-1 edge feature images are extracted for each of the N-1 other candidate images; the resolution of the edge feature images is the first target resolution;

[0071] Different candidate images correspond to different preset convolution kernels; the preset convolution kernels are used to extract gradient feature information between pixels in a preset direction; the preset direction is the relative direction of different pixels at the same position in the same pixel matrix between the corresponding candidate image and the starting image.

[0072] Optionally, the first integration unit is used for:

[0073] The N candidate images obtained from the segmentation and the N-1 edge feature images extracted are input into a pre-trained image channel merging network to obtain a target image with a resolution of the first target resolution output by the image channel merging network.

[0074] Optionally, the image channel merging network is used for:

[0075] For the N-1 other candidate images, each image is superimposed with its corresponding edge feature image to obtain N-1 superimposed images;

[0076] For the N-1 superimposed images and the starting image, features are extracted using the representation layer to obtain N convolutional feature images;

[0077] The N convolutional feature images are superimposed on each other by channels, and then the channel merging layer is used to extract features to obtain a target image with the same resolution as the first target resolution and the same number of channels as the image to be adjusted.

[0078] Optionally, the first feature unit is used for:

[0079] For each identical location among the N candidate images obtained from the segmentation, perform the following operations to obtain a max-pooling feature map with a resolution of the first target resolution, and determine the max-pooling feature map as the extracted edge feature information:

[0080] The pixel value with the largest value among the pixel values ​​of the N pixels at the target position in the N candidate images is determined as the pixel value of the pixel at the target position in the max pooling feature map.

[0081] Optionally, the first integration unit is used for:

[0082] The N candidate images obtained from the segmentation and the max pooling feature map are input into a pre-trained image channel fusion network to obtain a target image with a resolution of the first target resolution output by the image channel fusion network.

[0083] Optionally, the image channel fusion network is used for:

[0084] For the N candidate images, each image is superimposed with the max pooling feature map through channels to obtain N superimposed results;

[0085] For the N superimposed results, features are extracted using the representation layer to obtain N alternative images with the same resolution as the first target resolution and the same number of channels as the image to be adjusted.

[0086] The N candidate images are input into a max pooling layer to obtain a target image with the same resolution as the first target resolution and the same number of channels as the image to be adjusted.

[0087] Optionally, the max pooling layer is used for:

[0088] For each identical position among the N candidate images, perform the following operations to obtain a target image with the same resolution as the first target resolution and the same number of channels as the image to be adjusted:

[0089] The pixel value with the largest value among the pixel values ​​of the N pixels at the target position in the N candidate images is determined as the pixel value of the pixel at the target position in the target image.

[0090] According to a fourth aspect of the present invention, another image resolution adjustment apparatus is provided, comprising:

[0091] The second acquisition unit is used to acquire the image to be adjusted and the second target resolution; the resolution of the image to be adjusted is lower than the second target resolution;

[0092] The direction determination unit is used to determine M pixel expansion directions based on the second target resolution and the resolution of the image to be adjusted; M is a positive integer and M≥1; wherein, the product of (M+1) and the resolution of the image to be adjusted is greater than or equal to the second target resolution;

[0093] The second feature unit is used to acquire M edge feature information in the M pixel expansion directions for the image to be adjusted;

[0094] The second integration unit is used to integrate the image to be adjusted and the M edge feature information to obtain a target image with the resolution of the second target resolution.

[0095] Optionally, the second feature unit is used for:

[0096] For the image to be adjusted, M edge feature images are extracted using M preset convolution kernels corresponding to the M pixel expansion directions; the resolution of the edge feature images is the same as the resolution of the image to be adjusted.

[0097] Different pixel expansion directions correspond to different preset convolution kernels; the preset convolution kernels are used to extract the gradient feature information between pixels in the corresponding pixel expansion direction.

[0098] Optionally, the second integration unit is used for:

[0099] The image to be adjusted and the M edge feature images are input into a pre-trained image combination network to obtain a target image with the resolution of the second target resolution output by the image combination network; the target image has the same number of channels as the image to be adjusted.

[0100] Optionally, the image combining network is used for:

[0101] For the M edge feature images, each is superimposed with the image to be adjusted through channels to obtain M superimposed results;

[0102] For the M superposition results and the image to be adjusted, features are extracted using the representation layer to obtain M+1 images to be combined with the same resolution and number of channels as the image to be adjusted;

[0103] The M+1 images to be combined are input into the combination layer to obtain a target image with the second target resolution and the same number of channels as the images to be adjusted.

[0104] Optionally, the combined layer is used for:

[0105] Perform the following operations on each pixel in the image to be combined corresponding to the image to be adjusted, to obtain the combined result:

[0106] Starting from the target pixel, a pixel matrix containing M+1 pixels is obtained by expanding based on the M pixel expansion directions, and the pixels in the other M images to be combined that are at the same position as the target pixel are added to the expanded pixel matrix.

[0107] Based on the obtained combination results, a target image with the second target resolution and the same number of channels as the image to be adjusted is obtained through preset image processing.

[0108] As can be seen from the above embodiments, by introducing edge feature information for image resolution adjustment during the image resolution adjustment process, the display effect can be improved.

[0109] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0110] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0111] Figure 1 This is a schematic flowchart illustrating an image resolution adjustment method according to an embodiment of the present invention;

[0112] Figure 2 This is a schematic flowchart illustrating another image resolution adjustment method according to an embodiment of the present invention;

[0113] Figure 3 This is a schematic diagram of the structure of an image downsampling network according to an embodiment of the present invention;

[0114] Figure 4 This is a schematic diagram of another image downsampling network structure according to an embodiment of the present invention;

[0115] Figure 5 This is a schematic diagram of the structure of an image super-resolution network according to an embodiment of the present invention;

[0116] Figure 6 This is a schematic diagram of an image resolution adjustment device according to an embodiment of the present invention;

[0117] Figure 7This is a schematic diagram of another image resolution adjustment device according to an embodiment of the present invention;

[0118] Figure 8 A schematic diagram of the hardware structure of a computer device configured according to an embodiment of the present invention is shown. Detailed Implementation

[0119] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0120] In practical image processing, it is often necessary to adjust the image resolution. For example, for display screens of different sizes, it is usually necessary to adjust the resolution of a fixed-size image to fit the display screen. Specifically, this can be done by lowering or raising the image resolution.

[0121] In related technologies, some pixels in an image can be directly deleted to reduce the image resolution, or some pixels in an image can be directly copied to increase the image resolution.

[0122] For example, when reducing image resolution, simply delete pixels in odd-numbered rows or columns; when increasing image resolution, simply copy pixels in odd-numbered rows or columns.

[0123] However, using these methods to adjust image resolution often results in poor display quality after adjustment.

[0124] For example, when reducing image resolution, directly deleting some pixels results in the loss of some information, leading to lower image clarity and poor display quality; when increasing image resolution, directly copying some pixels and adding them to the image causes image distortion and poor display quality.

[0125] To improve the display effect of images after resolution adjustment, this invention provides an image resolution adjustment method.

[0126] In this method, edge feature information can be extracted from the image to be adjusted, which can then be used to adjust the image resolution. Specifically, this can involve increasing or decreasing the resolution.

[0127] To address the need for resolution reduction adjustments, this method preserves all pixels in the image to be adjusted, thereby reducing information loss and improving display quality.

[0128] To address the need for resolution enhancement, this method utilizes edge feature information in various expandable directions of the image to help expand pixels, thereby increasing resolution, reducing image distortion, and improving display quality.

[0129] Therefore, in this method, edge feature information can be introduced during the image resolution adjustment process, thereby improving the display effect. Compared to the simple and crude method of deleting and copying pixels for image resolution adjustment, this method can significantly improve the display effect.

[0130] The following is a detailed explanation of an image resolution adjustment method provided by an embodiment of the present invention.

[0131] This invention provides a method embodiment for reducing image resolution.

[0132] like Figure 1 As shown, Figure 1 This is a flowchart illustrating an image resolution adjustment method according to an embodiment of the present invention.

[0133] The embodiments of the present invention do not limit the executing entity of this method flow. Optionally, the executing entity can be any computing device, such as a server, display terminal, personal computer, camera, etc.

[0134] The method may include the following steps.

[0135] S101: Obtain the image to be adjusted and the first target resolution; the resolution of the image to be adjusted is higher than the first target resolution.

[0136] S102: Divide the image to be adjusted into N candidate images with a resolution of the first target resolution; wherein, the information of each pixel in the image to be adjusted can be contained in any candidate image; N is a positive integer and N>1.

[0137] S103: Extract edge feature information from the segmented candidate images.

[0138] S104: Combine the N candidate images obtained from the segmentation with the extracted edge feature information to obtain a target image with the resolution of the first target.

[0139] The above method can improve the display effect by introducing edge feature information during the image resolution adjustment process.

[0140] Furthermore, in the process of reducing resolution, the above method divides the image to be adjusted into multiple candidate images with the first target resolution, which facilitates subsequent feature extraction directly based on multiple candidate images. Specifically, it can be a feature extraction method that does not change the image resolution.

[0141] Furthermore, in the above method, by limiting each pixel in the image to be adjusted to be included in any candidate image, the resolution can be adjusted using all the pixels in the image to be adjusted, reducing the loss of information in the image to be adjusted and improving the display effect.

[0142] Alternatively, the above method can be executed entirely by a pre-trained image downsampling network. See the explanation below for details.

[0143] The following is a detailed explanation of each step.

[0144] 1. S101: Obtain the image to be adjusted and the first target resolution; the resolution of the image to be adjusted is higher than the first target resolution.

[0145] This method does not limit the format of the image to be adjusted. Specifically, it does not limit the number of channels of the image to be adjusted.

[0146] Optionally, the image to be adjusted may include an image with RGB three channels, or it may include a feature image with multiple channels.

[0147] Optionally, the image to be adjusted may include a feature image obtained by feature extraction based on the original image. The method of feature extraction is not limited, nor is the number of channels in the image to be adjusted limited.

[0148] Optionally, the first target resolution can be the target resolution that needs to be adjusted for the image to be adjusted.

[0149] 2. S102: Divide the image to be adjusted into N candidate images with a resolution of the first target resolution; wherein, the information of each pixel in the image to be adjusted can be contained in any candidate image; N is a positive integer and N>1.

[0150] This method does not limit how N is determined.

[0151] Optionally, the product of N and the first target resolution can be greater than or equal to the resolution of the image to be adjusted.

[0152] This method does not limit the way the data is divided.

[0153] Alternatively, to ensure the integrity of the candidate image, some pixels can be selected from the image to be adjusted and combined to form the candidate image; alternatively, the image to be adjusted can be divided into different parts.

[0154] Optionally, when the image to be adjusted is divided into multiple different parts, they can be combined and stitched together during subsequent integration to obtain the target image.

[0155] Optionally, overlapping portions are allowed between the different candidate images obtained from the segmentation. However, since the resolution of the image to be adjusted may not be an integer multiple of the first target resolution, it is not possible to directly segment N non-overlapping candidate images.

[0156] In one alternative embodiment, multiple pixel matrices can be divided first, and then a pixel can be selected from each pixel matrix to form a candidate image.

[0157] Optionally, dividing the image to be adjusted into N candidate images with a resolution of the first target resolution may include: dividing the image to be adjusted into several pixel matrices of the same size according to the first target resolution; the pixel matrix may include N pixels; and extracting N candidate images with a resolution of the first target resolution according to the divided pixel matrix.

[0158] Optionally, in the image to be adjusted, the number of single-row pixel matrices divided horizontally can be the same as the number of single-row pixels at the first target resolution, and the number of single-column pixel matrices divided vertically can be the same as the number of single-column pixels at the first target resolution. Subsequent steps can then synthesize each pixel matrix to obtain a single pixel, thereby obtaining a target image with the resolution of the first target resolution.

[0159] Optionally, extracting N candidate images with a resolution of the first target resolution based on the divided pixel matrix may include: extracting one pixel from each of the divided pixel matrices; and combining the extracted pixels into a candidate image according to the relative positional relationship between the pixel matrices.

[0160] Optionally, different candidate images can be obtained by extracting different pixels from the same pixel matrix, thus obtaining N candidate images.

[0161] Optionally, for a single candidate image, pixels can be extracted from the same position in each pixel matrix and combined to form a candidate image. Optionally, it is also possible to extract pixels from different positions in different pixel matrices and combine them to form a candidate image.

[0162] Optionally, pixels at the same position in different candidate images may belong to the same pixel matrix; different pixels in any candidate image may be located at the same position in their respective pixel matrices.

[0163] Optionally, overlapping portions are allowed between the divided pixel matrices.

[0164] In this embodiment, by dividing the pixel matrix, it is easy to divide the candidate image with high integrity, which facilitates the subsequent extraction of edge feature information.

[0165] To facilitate understanding, in a specific example, for an image to be adjusted with a resolution of 16*16, the first target resolution can be 8*8. This allows the image to be divided into 64 non-overlapping 2*2 pixel matrices, making it easier to extract four candidate images with a resolution of 8*8.

[0166] 3. S103: Extract edge feature information from the segmented candidate images. S104: Combine the N segmented candidate images and the extracted edge feature information to obtain a target image with the first target resolution.

[0167] Since the extracted edge feature information is related to the subsequent integration method, the merging of S103 and S104 is explained.

[0168] This method does not limit the specific way of extracting edge feature information.

[0169] Optionally, edge feature information can be extracted for each candidate image separately; edge feature information can be extracted for some candidate images separately; or all candidate images can be combined and edge feature information can be extracted using pooling, specifically max pooling, average pooling, or min pooling.

[0170] This methodology does not limit the specific synthesis method.

[0171] Optionally, a pre-trained network or model can be used to input N candidate images and the extracted edge feature information to obtain a target image with a resolution of the first target resolution output by the network or model; alternatively, the N candidate images and the extracted edge feature information can be directly superimposed on each channel, and the channels can be merged based on the superposition result to obtain a target image with a resolution of the first target resolution.

[0172] For ease of understanding, two optional embodiments are provided below for illustrative purposes.

[0173] Example 1.

[0174] In this embodiment, dividing the image to be adjusted into N candidate images with a resolution of the first target resolution may include: dividing the image to be adjusted into several pixel matrices of the same size according to the first target resolution; the pixel matrix may include N pixels; and extracting N candidate images with a resolution of the first target resolution according to the divided pixel matrix.

[0175] In the image to be adjusted, the number of single-row pixel matrices in the horizontal direction can be the same as the number of single-row pixels in the first target resolution, and the number of single-column pixel matrices in the vertical direction can be the same as the number of single-column pixels in the first target resolution.

[0176] Pixels at the same position in different candidate images can belong to the same pixel matrix; different pixels in any candidate image can be located at the same position in their respective pixel matrices.

[0177] Furthermore, in this embodiment, any candidate image can be determined as the starting image, thereby determining the same pixel matrix to which two pixels at the same position belong between any other candidate image and the starting image, and determining the fixed relative direction between the two pixels in the same pixel matrix.

[0178] Based on this relative direction, the inter-pixel gradient feature information of the starting image in this relative direction can be obtained to characterize edge features. Furthermore, the extracted inter-pixel gradient feature information, along with other corresponding candidate images, can be used to characterize the changes in pixels in the image to be adjusted, thus facilitating feature extraction for resolution adjustment and improving the display effect.

[0179] Optionally, extracting edge feature information for the segmented candidate images may include: determining any candidate image as the starting image; for the starting image, using N-1 preset convolutional kernels corresponding to N-1 other candidate images to extract N-1 edge feature images corresponding to each of the N-1 other candidate images; the resolution of the edge feature images may be the first target resolution.

[0180] Optionally, different candidate images may correspond to different preset convolution kernels; the preset convolution kernels may be used to extract gradient feature information between pixels in a preset direction; the preset direction may include the relative direction of different pixels at the same position in the same pixel matrix between the corresponding candidate image and the starting image.

[0181] Examples of specific pre-defined convolutional kernels can be found below.

[0182] This embodiment does not limit the synthesis method.

[0183] Optionally, the N candidate images obtained from the segmentation and the N-1 edge feature images extracted can be overlaid with channels, and then the features can be extracted to obtain the target image.

[0184] Optionally, N-1 other candidate images can be overlaid with the corresponding N-1 edge feature images to obtain N-1 overlay results. Then, features are extracted from the N-1 overlay results and the starting image respectively. The extracted N features are combined to obtain the target image.

[0185] Alternatively, a pre-trained model or network can be used to extract features from the input N candidate images and the extracted N-1 edge feature images to obtain the target image.

[0186] Optionally, combining the N candidate images obtained from the segmentation and the extracted edge feature information to obtain a target image with a resolution of the first target resolution may include: inputting the N candidate images obtained from the segmentation and the N-1 edge feature images extracted into a pre-trained image channel merging network to obtain a target image with a resolution of the first target resolution output by the image channel merging network.

[0187] Optionally, the target image and the image to be adjusted can have the same number of channels.

[0188] This embodiment does not limit the specific structure of the image channel merging network.

[0189] Optionally, the image channel merging network may include only convolutional layers, or it may include both convolutional layers and an output layer to output the target image.

[0190] Optionally, the image channel merging network may include a representation layer and a channel merging layer. The representation layer can be used to extract image features, and may specifically include one or more convolutional layers. The channel merging layer can be used to extract image features and merge image channels, reducing the number of image channels.

[0191] Optionally, the image channel merging network can be used to: overlay the corresponding edge feature images with N-1 other candidate images to obtain N-1 overlaid images; extract features from the N-1 overlaid images and the starting image using the representation layer to obtain N convolutional feature images; overlay the N convolutional feature images with channels, and then extract features using the channel merging layer to obtain a target image with a resolution of the first target resolution and the same number of channels as the image to be adjusted.

[0192] Optionally, the image channel merging network can also be used to: extract features from N candidate images and N-1 edge feature images using the representation layer to obtain 2N-1 convolutional feature images; stack the 2N-1 convolutional feature images by channels, and then extract features using the channel merging layer to obtain a target image with the first target resolution and the same number of channels as the image to be adjusted.

[0193] This embodiment does not limit the structure and function of the characterization layer and the channel merging layer.

[0194] Optionally, the representation layer may include multiple different convolutional layers, specifically N different convolutional layers. Thus, features can be extracted from N-1 stacked images and the initial image using different convolutional layers in the representation layer, resulting in N convolutional feature images.

[0195] Optionally, the number of channels in the N-1 stacked images is usually greater than that in the initial image. Therefore, the convolutional layers corresponding to the N-1 stacked images can be used to merge image channels, thereby reducing the number of image channels. Optionally, the number of channels in the N convolutional feature images can be the same.

[0196] By using different convolutional layers, image features from different branches can be learned separately, which facilitates the improvement of the image channel merging network and enhances the display effect.

[0197] Optionally, the channel merging layer may include one or more cascaded convolutional layers, thereby reducing the number of image channels by merging image channels through feature extraction.

[0198] Example 2.

[0199] In this embodiment, dividing the image to be adjusted into N candidate images with a resolution of the first target resolution may include: dividing the image to be adjusted into several pixel matrices of the same size according to the first target resolution; the pixel matrix may include N pixels; and extracting N candidate images with a resolution of the first target resolution according to the divided pixel matrix.

[0200] In the image to be adjusted, the number of single-row pixel matrices in the horizontal direction can be the same as the number of single-row pixels in the first target resolution, and the number of single-column pixel matrices in the vertical direction can be the same as the number of single-column pixels in the first target resolution.

[0201] Optionally, pixels at the same position in different candidate images may belong to the same pixel matrix; different pixels in any candidate image may be located at the same position in their respective pixel matrices.

[0202] Optionally, different pixels in any candidate image can also be located at different positions in their respective pixel matrix. This allows different pixels in a single candidate image to be located at different positions in their respective pixel matrix.

[0203] Furthermore, in this embodiment, max pooling can be used to extract edge feature information of the image, which facilitates the extraction of texture details in the image, thereby facilitating feature extraction for resolution adjustment and improving the display effect.

[0204] Optionally, for the segmented candidate images, extracting edge feature information may include: for each common position among the segmented N candidate images, performing the following operations to obtain a max pooling feature map with a resolution of the first target resolution, and determining the max pooling feature map as the extracted edge feature information: determining the largest pixel value among the pixel values ​​corresponding to the N pixels at the targeted position in the N candidate images as the pixel value corresponding to the pixel at the targeted position in the max pooling feature map.

[0205] This embodiment does not limit the synthesis method.

[0206] Optionally, the N candidate images obtained from the segmentation and the max pooling feature map can be overlaid with channels, and then the features can be extracted to obtain the target image.

[0207] Optionally, N candidate images can be superimposed with the max pooling feature map to obtain N superimposed results. Features can then be extracted from each of the N superimposed results. The extracted N features can be combined to obtain the target image.

[0208] Alternatively, a pre-trained model or network can be used to extract features from the input N candidate images and max-pooling feature maps to obtain the target image.

[0209] Optionally, combining the N candidate images obtained from the segmentation and the extracted edge feature information to obtain a target image with a resolution of the first target resolution may include: inputting the N candidate images obtained from the segmentation and the max pooling feature map into a pre-trained image channel fusion network to obtain a target image with a resolution of the first target resolution output by the image channel fusion network.

[0210] Optionally, the target image can have the same number of channels as the image to be adjusted.

[0211] This embodiment does not limit the specific structure of the image channel fusion network.

[0212] Optionally, the image channel fusion network may include only convolutional layers, or it may include both convolutional layers and an output layer to output the target image.

[0213] Optionally, the image channel fusion network may include a representation layer and a max-pooling layer. The representation layer can be used to extract image features and may specifically include one or more convolutional layers. The max-pooling layer can be used to synthesize multiple images output from the representation layer using max pooling to obtain the target image.

[0214] Optionally, the image channel fusion network can be used to: for N candidate images, perform channel superposition with the max pooling feature map respectively to obtain N superposition results; for the N superposition results, extract features using the representation layer respectively to obtain N candidate images with a resolution of the first target resolution and the same number of channels as the image to be adjusted; input the N candidate images into the max pooling layer to obtain a target image with a resolution of the first target resolution and the same number of channels as the image to be adjusted.

[0215] Optionally, the image channel fusion network can also be used to: extract features from N candidate images and max pooling feature maps using the representation layer to obtain N+1 candidate images with the first target resolution and the same number of channels as the image to be adjusted; input the N+1 candidate images into the max pooling layer to obtain the target image with the first target resolution and the same number of channels as the image to be adjusted.

[0216] This embodiment does not limit the structure and function of the characterization layer and the maximum pooling layer.

[0217] Optionally, the representation layer may include multiple different convolutional layers, specifically N different convolutional layers. Thus, for the N stacked results, features can be extracted using different convolutional layers in the representation layer to obtain N candidate images. Specifically, a feature extraction method that does not change the image resolution can be used.

[0218] Optionally, the representation layer can perform channel merging on each of the N overlay results to reduce the number of image channels.

[0219] By using different convolutional layers, image features from different branches can be learned separately, which facilitates the improvement of the image channel merging network and enhances the display effect.

[0220] Optionally, the max pooling layer can be used to: for each identical position among N candidate images, perform the following operations to obtain a target image with a resolution of the first target resolution: determine the largest pixel value among the pixel values ​​corresponding to the N pixels at the targeted position in the N candidate images as the pixel value corresponding to the pixel at the targeted position in the target image.

[0221] In another optional embodiment, the target images obtained from Embodiment 1 and Embodiment 2 can be combined to obtain the final target image.

[0222] In one alternative embodiment, the above method flow includes embodiments that can be implemented using a single, monolithic image downsampling network.

[0223] Image downsampling networks can be used to downsample an image to be adjusted, thereby reducing its resolution.

[0224] Optionally, the image to be adjusted and the first target resolution can be input into the image downsampling network to obtain the target image output by the image downsampling network.

[0225] Optionally, the image downsampling network may include the image channel merging network and / or image channel fusion network described in the above embodiments. The image downsampling network may further execute the methods of Embodiment 1 and / or Embodiment 2 described above on the N candidate images obtained by dividing the image to be adjusted to obtain the target image.

[0226] The above method flow explains an embodiment of reducing resolution.

[0227] This invention also provides an embodiment of a method for improving image resolution.

[0228] like Figure 2 As shown, Figure 2 This is a flowchart illustrating another image resolution adjustment method according to an embodiment of the present invention.

[0229] The embodiments of the present invention do not limit the executing entity of this method flow. Optionally, the executing entity can be any computing device, such as a server, display terminal, personal computer, camera, etc.

[0230] The method may include the following steps.

[0231] S201: Obtain the image to be adjusted and the resolution of the second target; the resolution of the image to be adjusted is lower than the resolution of the second target.

[0232] S202: Determine the M pixel expansion directions based on the second target resolution and the resolution of the image to be adjusted.

[0233] Optionally, M can be a positive integer and M≥1; where the product of (M+1) and the resolution of the image to be adjusted can be greater than or equal to the second target resolution.

[0234] S203: For the image to be adjusted, obtain M edge feature information in the M pixel expansion directions respectively.

[0235] S204: By combining the image to be adjusted and M edge feature information, a target image with a resolution of the second target resolution is obtained.

[0236] The above method can improve the display effect by introducing edge feature information during the image resolution adjustment process.

[0237] Furthermore, the above method can determine multiple pixel expansion directions during the resolution improvement process, and can also determine the edge feature information of each pixel expansion direction, thereby facilitating subsequent feature extraction and improving the display effect.

[0238] Alternatively, the above method can be executed entirely by a pre-trained image super-resolution network. See the explanation below for details.

[0239] The following is a detailed explanation of each step.

[0240] 1. S201: Obtain the resolution of the image to be adjusted and the resolution of the second target; the resolution of the image to be adjusted is lower than the resolution of the second target.

[0241] This method does not limit the format of the image to be adjusted. Specifically, it does not limit the number of channels of the image to be adjusted.

[0242] Optionally, the image to be adjusted may include an image with RGB three channels, or it may include a feature image with multiple channels.

[0243] Optionally, the image to be adjusted may include a feature image obtained by feature extraction based on the original image. The method of feature extraction is not limited, nor is the number of channels in the image to be adjusted limited.

[0244] Optionally, the second target resolution can be the target resolution that needs to be adjusted for the image to be adjusted.

[0245] 2. S202: Determine the expansion direction of M pixels based on the resolution of the second target and the resolution of the image to be adjusted.

[0246] This method does not limit the way M is determined, nor does it limit the way the pixel expansion direction is determined.

[0247] Optionally, M can be a positive integer and M≥1; where the product of (M+1) and the resolution of the image to be adjusted can be greater than or equal to the second target resolution.

[0248] Optionally, the pixel expansion direction can be determined based on the size relationship between the width and height of the second target resolution and the resolution of the image to be adjusted.

[0249] For example, given an image to be adjusted with a resolution of 4x4 (width and height both 4), and a second target resolution of 4x8 (width 4, height 8), it can be determined that pixel expansion needs to be performed vertically to achieve image super-resolution, which means increasing the image resolution. Therefore, the vertical direction can be determined as the pixel expansion direction.

[0250] For example, given an image to be adjusted with a resolution of 4x4 (both width and height are 4), and a second target resolution of 8x8 (width and height are 8), it can be determined that pixel expansion needs to be performed in three directions. Specifically, this can include vertical, horizontal, and diagonal directions.

[0251] 3. S203: For the image to be adjusted, acquire M edge feature information along the M pixel expansion directions respectively. S204: Combine the image to be adjusted and the M edge feature information to obtain a target image with a resolution of the second target resolution.

[0252] Since the extracted edge feature information is related to the subsequent integration method, the merging of S203 and S204 is explained.

[0253] This method does not limit the specific way of extracting edge feature information.

[0254] Optionally, gradient feature information between pixels in the direction of pixel expansion can be extracted; or pixel value change feature information in the direction of pixel expansion can be extracted.

[0255] This methodology does not limit the specific synthesis method.

[0256] Optionally, a pre-trained network or model can be used, with the image to be adjusted and M edge feature information as input, to obtain a target image with a resolution of the second target resolution output by the network or model; alternatively, the image to be adjusted can be superimposed with the M edge feature information channels respectively, the channels can be merged based on the superposition result, and then the images can be combined to obtain a target image with a resolution of the second target resolution.

[0257] For ease of understanding, an optional embodiment is provided below for illustrative purposes.

[0258] Optionally, for the image to be adjusted, obtaining M edge feature information in the M pixel expansion directions can include: for the image to be adjusted, extracting M edge feature images using M preset convolution kernels corresponding to the M pixel expansion directions; the resolution of the edge feature images is the same as the resolution of the image to be adjusted.

[0259] Optionally, different pixel expansion directions can correspond to different preset convolution kernels; the preset convolution kernels can be used to extract inter-pixel gradient feature information in the corresponding pixel expansion direction.

[0260] Specific examples of preset convolutional kernels can be found below.

[0261] This embodiment does not limit the synthesis method.

[0262] Alternatively, the M edge feature images and the image to be adjusted can be directly combined to obtain the target image.

[0263] Optionally, feature images can be extracted from the M edge feature images and the image to be adjusted respectively, and then combined to obtain the target image.

[0264] Optionally, a pre-trained model or network can be used to extract features from the input M edge feature images and the image to be adjusted to obtain the target image.

[0265] Optionally, combining the image to be adjusted and M edge feature information to obtain a target image with a resolution of the second target resolution may include: inputting the image to be adjusted and the M edge feature images into a pre-trained image combination network to obtain a target image with a resolution of the second target resolution output by the image combination network; the target image and the image to be adjusted have the same number of channels.

[0266] This embodiment does not limit the specific structure of the image combination network.

[0267] Optionally, the image compositing network may include only convolutional layers, or it may include both convolutional layers and an output layer to output the target image.

[0268] Optionally, the image ensemble network may include a representation layer and a ensemble layer. The representation layer can be used to extract image features, and may specifically include one or more convolutional layers, or a UNet network structure.

[0269] Optionally, the representation layer can be used for feature focusing, specifically for focusing features on the image to be adjusted and the M edge feature images respectively, thereby learning the image features of different branches and improving the display effect of image super-resolution.

[0270] Optionally, the combination layer can be used to combine the images output by the representation layer and output a target image with a resolution of the second target resolution.

[0271] Alternatively, the combined layer may not change the number of channels in the input and output images.

[0272] Optionally, the image combination network can be used to: superimpose M edge feature images with the image to be adjusted through channels to obtain M superimposed results; extract features from the M superimposed results and the image to be adjusted using the representation layer to obtain M+1 images to be combined with the same resolution and number of channels as the image to be adjusted; and input the M+1 images to be combined into the combination layer to obtain a target image with the second target resolution and the same number of channels as the image to be adjusted.

[0273] Of course, alternatively, the image combination network may not perform channel stacking, but can directly extract features from M edge feature images and the image to be adjusted using the representation layer to obtain M+1 images to be combined with the same resolution and number of channels as the image to be adjusted; the M+1 images to be combined are then input into the combination layer to obtain a target image with the second target resolution and the same number of channels as the image to be adjusted.

[0274] This embodiment does not limit the structure and function of the characterization layer and the combination layer.

[0275] Optionally, the representation layer may include multiple different convolutional layers, specifically M+1 different convolutional layers. Thus, features can be extracted from the M stacked results and the image to be adjusted using different convolutional layers in the representation layer, resulting in M+1 images to be combined.

[0276] Optionally, the representation layer may include multiple different feature-focusing layers, and the input and output images of the feature-focusing layers may have the same resolution and number of channels. Specifically, the feature-focusing layer may include a Unet network. The representation layer may specifically include M+1 different feature-focusing layers. Thus, for M stacked results and the image to be adjusted, features can be extracted using different feature-focusing layers in the representation layer to obtain M+1 images to be combined.

[0277] By using different convolutional layers and / or feature-focusing layers, image features of different branches can be learned separately, thereby facilitating the improvement of the image combination network's performance and enhancing the display effect.

[0278] Optionally, the combination layer can be used to perform the following operations for each pixel in the image to be combined corresponding to the image to be adjusted, to obtain the combination result: starting from the target pixel, expand it in M ​​pixel expansion directions to obtain a pixel matrix containing M+1 pixels, and add the pixels in the other M images to be combined that are at the same position as the target pixel to the expanded pixel matrix; based on the obtained combination result, obtain a target image with a resolution of the second target resolution and the same number of channels as the image to be adjusted after preset image processing.

[0279] Optionally, the expanded pixel matrix may include the starting pixel.

[0280] Optionally, since the product of (M+1) and the resolution of the image to be adjusted can be greater than or equal to the second target resolution, the resolution of the resulting combination may be greater than or equal to the second target resolution.

[0281] This embodiment does not limit the preset image processing method.

[0282] Alternatively, if the resolution of the combined result is equal to the resolution of the second target, the combined result can be directly determined as the target image.

[0283] Optionally, if the resolution of the combined result is greater than the second target resolution, a target image with the same resolution as the image to be adjusted can be obtained by deleting or merging some pixels, or by image downsampling, such as in the embodiments of the above method flow S101-S104.

[0284] In one alternative embodiment, the above method flow includes embodiments that can be implemented using a single, monolithic image super-resolution network.

[0285] Image super-resolution networks can be used to super-resolution images to improve their resolution.

[0286] Optionally, the image to be adjusted and the second target resolution can be input into the image super-resolution network to obtain the target image output by the image super-resolution network.

[0287] Optionally, the image super-resolution network may include the image combination network described in the above embodiments. The image super-resolution network can determine M pixel expansion directions for the image to be adjusted, and then execute the method described in the above embodiments to obtain the target image.

[0288] For ease of understanding, the present invention also provides three application embodiments.

[0289] Application Example 1.

[0290] like Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of an image downsampling network according to an embodiment of the present invention.

[0291] The image downsampling network can include a split layer (mux layer), a directional operator layer, a convolutional layer, and an output layer.

[0292] Figure 3 The image downsampling network in the image can reduce the resolution of an image with a resolution of 20*20 to 10*10.

[0293] The mux layer decomposes a two-dimensional matrix into four smaller matrices, each half the length and width of the original matrix, according to the pixel arrangement rules shown in the diagram. Figure 3 (M1, M2, M3, and M4 in the text).

[0294] For an input image to be adjusted (with nf channels), an image downsampling network can use a mux layer to split it into four 10*10 candidate images. Specifically, the image to be adjusted can be divided into 100 2*2 pixel matrices, with each pixel matrix having a11, b11, c11, and d11.

[0295] Then, a pixel can be selected from a fixed position in each pixel matrix to form a candidate image. For example, the top-left pixel of each 2x2 pixel matrix can be selected to form a candidate image including pixels a11, a12, ... a21, a22, etc. (hereinafter, candidate image a is the one referred to as 'a'). Figure 3 (M1 in the middle).

[0296] For the direction operator layer, one of the candidate images can be selected as the starting image. Figure 3 The candidate image 'a' was selected as the starting image, which allows us to determine the relative orientation of pixels at the same position in the pixel matrix of each candidate image.

[0297] For example, a candidate image including pixels b11, b12, ... b21, b22, etc. (hereinafter referred to as candidate image b) Figure 3 In M2), b11 is to the right of a11.

[0298] Furthermore, the gradient feature information between pixels in the candidate image a (i.e., M1) in the relative direction can be extracted by using the preset convolution kernels corresponding to the relative directions.

[0299] For example, for a preset convolution kernel corresponding to the right direction, the gradient feature information between pixels in the right direction can be extracted.

[0300] certainly, Figure 3 The example of the preset convolution kernels in this specification is for illustrative purposes only and does not limit the scope of this specification.

[0301] In the orientation operator layer, for the same relative direction, the "inter-pixel gradient feature information of candidate image 'a' in that direction" and "other candidate images in that direction" can be superimposed using channels for subsequent feature extraction by convolutional layers. The number of channels in the superimposed result is nf*2.

[0302] For example, for the direction operator in the right direction, the gradient feature information between pixels in the right direction of candidate image a (M1) and candidate image b can be superimposed on the channel.

[0303] For convolutional layers, different convolutional layers can be used to extract features for the superposition results of the candidate image (M1) and the outputs of the directional operator layers in the other three branches.

[0304] For candidate image 'a', features can be extracted directly using a convolutional layer with the same number of channels. Alternatively, feature extraction can be omitted.

[0305] For the superimposed results in the other three branches, features can be extracted using a convolutional layer that merges image channels, reducing the number of channels from nf*2 to nf.

[0306] Figure 3 In this context, Conv(nf*2->nf) represents the number of channels in this convolutional layer. The input is nf*2 and the output is nf. (nf can be 64, 48, 32, etc.)

[0307] For the output layer, the features extracted by the convolutional layers in the four branches can be stacked by channels, and then the stacked result can be fused by channels through the convolutional layer. The resolution can remain unchanged, so as to obtain a target image with the first target resolution and the same number of channels as the image to be adjusted.

[0308] After the subsequent Concat channel stacking, the four sets of features can be fused through a convolutional layer with nf*4 input channels and nf output channels.

[0309] This network structure can be used as a downsampling operator in deep learning networks.

[0310] Application Example 2.

[0311] like Figure 4 As shown, Figure 4 This is a schematic diagram of another image downsampling network structure according to an embodiment of the present invention.

[0312] The image downsampling network can include a splitting layer (mux layer), a downsampling max pooling layer (max pooling layer), a convolutional layer, and an inter-image max pooling layer (Gmax pooling layer).

[0313] Figure 4 The image downsampling network in the model can reduce the resolution of an image to be adjusted from 20*20 (with nf channels) to 10*10.

[0314] The mux layer can be found in the explanation of the above application example one.

[0315] For downsampling max pooling layers, it is possible to determine the largest pixel value in each partitioned pixel matrix, thereby obtaining a 10*10 max pooling feature map.

[0316] For example, for a pixel matrix (a11, b11, c11, and d11), the largest pixel value can be determined as the pixel value in the max-pooling feature map.

[0317] The Maxpooling layer can be an image obtained by downsampling the image to be adjusted (I) through max pooling. Specifically, it can retain the maximum value within a small region, which can better represent the detailed texture information of the image.

[0318] For the convolutional layer, the max pooling feature map can be stacked and convolved with the four candidate images through channels. Specifically, feature fusion can be performed through convolution to obtain four candidate images (F1-F4) with nf channels and a resolution of 10*10.

[0319] The M1, M2, M3, and M4 obtained after Mux can be superimposed with the max pooling feature map and then fused through convolution. Here, the input channel of Conv is nf*2 and the output channel is nf.

[0320] For the inter-image max pooling layer, inter-image max pooling can be performed on F1-F4, which means determining the maximum pixel value among four pixels at the same position between F1 and F4, thereby obtaining the final target image with nf channels and a resolution of 10*10.

[0321] The Gmaxpooling layer is used to take the maximum value of F1 / F2 / F3 / F4 point-to-point and output it. Gmaxpooling(F1,F2,F3,F4)=MAX(F1,F2,F3,F4).

[0322] Application Example 3.

[0323] like Figure 5 As shown, Figure 5 This is a schematic diagram of the structure of an image super-resolution network according to an embodiment of the present invention.

[0324] Based on the idea of ​​directional operator downsampling mentioned above, image super-resolution networks can be designed similarly.

[0325] The image super-resolution network can include a feature extraction layer, a direction operator layer, a feature focusing layer, and a combination layer (DeMux layer).

[0326] Figure 5 The image super-resolution network in the article can increase the resolution of an image to be adjusted from 10*10 (with nf channels) to 20*20.

[0327] For the feature extraction layer, a convolutional layer can be used to extract features from image I. The extracted feature map F can have the same resolution as image I, but the number of channels in feature map F can be greater than that in image I.

[0328] It should be noted that image I can be regarded as the image to be adjusted, or feature map F can be regarded as the image to be adjusted.

[0329] Figure 5 The purpose of the feature extraction layer in the network is to convert image information into feature information. The simplest method is to use a 3x3 convolutional layer, or you can use 2 / 3 / n convolutional layers in succession for feature extraction. This embodiment does not impose any specific network structure restrictions.

[0330] For the direction operator layer, three pixel expansion directions can be determined: right, down, and diagonally to the lower right.

[0331] First, let's explain the direction operators of each branch. For the feature map F, we can use a preset convolution kernel to extract the gradient feature information between pixels in the expansion direction of each pixel.

[0332] The extracted inter-pixel gradient feature information can then be superimposed on the feature map F.

[0333] Specifically, the gradient feature information between pixels extracted using a preset convolutional kernel can be a gradient feature map. The resolution of the gradient feature map can be the same as that of the feature map F.

[0334] For the feature focusing layer, this embodiment does not limit the specific structure. Optionally, convolutional layers or a UNet structure can be used to extract features.

[0335] Figure 5 The feature focusing layer in this algorithm extracts feature information from each branch of the image; for example, a typical UNet structure can be used. Similarly, UNet is just one example here; theoretically, any network commonly used in super-resolution can be used as a feature focusing layer here.

[0336] Each feature-focusing layer can extract features from the superposition results of each branch's output, resulting in four images to be combined. The resolution of the images to be combined is 10*10, and the number of channels is nf.

[0337] It should be noted that the feature map F itself can utilize the feature focusing layer, specifically by extracting features through Unet, while the number of channels and resolution of the input and output images can remain unchanged.

[0338] For the other three branches utilizing the direction operator, channel stacking yields an nf*2 stacked result, allowing the use of a feature focusing layer to extract feature maps with the same resolution but reduced channel count to nf. Optionally, in the other three branches utilizing the direction operator, a Conv(nf*2->nf) can be set before the feature focusing layer in Unet.

[0339] For the combination layer, the four images to be combined can be combined.

[0340] In previous application examples, the Mux layer can divide the pixels of an image into four small parts. Based on the position of the pixels, it can be found that each small part has certain directional information. Therefore, the information of the image can be extracted more effectively by using the directional operator.

[0341] Similarly, DeMux merges four small images into one large image according to the same arrangement rules as Mux. Therefore, when designing a super-resolution network, we can use the direction operator to divide it into four branches for feature extraction, and finally use DeMux to super-resolution the large image.

[0342] The Demux layer is used to combine two-dimensional matrices. It can combine four small matrices of the same size into a large matrix, where the length and width of the large matrix are twice that of the small matrices.

[0343] For example, pixels at the same position (a11, b11, c11, and d11) in four images to be combined can be combined into the same pixel matrix to obtain a target image with a resolution of 20*20.

[0344] The combination layer can obtain a target image with nf channels from an image to be combined with nf channels without changing the number of channels in the input and output images.

[0345] Corresponding to the above method embodiments, the present invention also provides corresponding device embodiments.

[0346] like Figure 6 As shown, Figure 6 This is a schematic diagram of an image resolution adjustment device according to an embodiment of the present invention.

[0347] The device may include the following units.

[0348] The first acquisition unit 301 is used to acquire the image to be adjusted and the first target resolution; the resolution of the image to be adjusted is higher than the first target resolution;

[0349] The segmentation unit 302 is used to divide the image to be adjusted into N candidate images with a resolution of the first target resolution; wherein the information of each pixel in the image to be adjusted is contained in any candidate image; N is a positive integer and N>1;

[0350] The first feature unit 303 is used to extract edge feature information from the segmented candidate image;

[0351] The first integration unit 304 is used to integrate the N candidate images obtained from the division and the extracted edge feature information to obtain a target image with a resolution of the first target resolution.

[0352] Optionally, the partitioning unit 302 is used for:

[0353] Based on the first target resolution, several pixel matrices of the same size are divided from the image to be adjusted; each pixel matrix contains N pixels.

[0354] In the image to be adjusted, the number of single-row pixel matrices divided horizontally is the same as the number of single-row pixels in the first target resolution, and the number of single-column pixel matrices divided vertically is the same as the number of single-column pixels in the first target resolution.

[0355] Based on the divided pixel matrix, N candidate images with a resolution equal to the first target resolution are extracted.

[0356] Optionally, pixels at the same position in different candidate images belong to the same pixel matrix; different pixels in any candidate image are located at the same position in their respective pixel matrices.

[0357] Optionally, the first feature unit 303 is used for:

[0358] Select any candidate image as the starting image;

[0359] For the initial image, N-1 preset convolutional kernels corresponding to N-1 other candidate images are used to extract N-1 edge feature images corresponding to each of the N-1 other candidate images; the resolution of the edge feature images is the first target resolution;

[0360] Different candidate images correspond to different preset convolution kernels; the preset convolution kernels are used to extract gradient feature information between pixels in a preset direction; the preset direction is the relative direction of different pixels at the same position in the same pixel matrix between the corresponding candidate image and the starting image.

[0361] Optionally, the first integration unit 304 is used for:

[0362] The N candidate images obtained from the segmentation and the N-1 extracted edge feature images are input into a pre-trained image channel merging network to obtain a target image with a resolution of the first target resolution output by the image channel merging network. Optionally, the target image can have the same number of channels as the image to be adjusted.

[0363] Optionally, the image channel merging network is used for:

[0364] For the N-1 other candidate images, each image is overlaid with its corresponding edge feature image to obtain N-1 overlaid images;

[0365] For N-1 superimposed images and the initial image, features are extracted using the representation layer to obtain N convolutional feature images;

[0366] The N convolutional feature images are stacked by channels, and then the channel merging layer is used to extract features to obtain a target image with the same resolution as the first target resolution and the same number of channels as the image to be adjusted.

[0367] Optionally, the first feature unit 303 is used for:

[0368] For each identical location among the N candidate images obtained from the segmentation, perform the following operations to obtain a max-pooling feature map with a resolution of the first target resolution, and determine the max-pooling feature map as the extracted edge feature information:

[0369] The pixel value with the largest value among the pixel values ​​of the N pixels at the target position in the N candidate images is determined as the pixel value of the pixel at the target position in the max pooling feature map.

[0370] Optionally, the first integration unit 304 is used for:

[0371] The N candidate images obtained from the segmentation and the max-pooling feature map are input into a pre-trained image channel fusion network to obtain a target image with a resolution of the first target resolution output by the image channel fusion network. Optionally, the target image can have the same number of channels as the image to be adjusted.

[0372] Optionally, the image channel fusion network is used for:

[0373] For N candidate images, each image is superimposed with a max pooling feature map through channels to obtain N superimposed results;

[0374] For each of the N overlay results, features are extracted using the representation layer to obtain N alternative images with the same resolution as the first target resolution and the same number of channels as the image to be adjusted.

[0375] Input N candidate images into a max pooling layer to obtain a target image with the same resolution as the first target resolution and the same number of channels as the image to be adjusted.

[0376] Optionally, the max pooling layer is used for:

[0377] For each identical location among the N candidate images, perform the following operations to obtain a target image with the same resolution as the first target resolution and the same number of channels as the image to be adjusted:

[0378] The largest pixel value among the pixel values ​​corresponding to the N pixels at the target position in the N candidate images is determined as the pixel value corresponding to the pixel at the target position in the target image.

[0379] For a detailed explanation, please refer to the above method implementation examples.

[0380] like Figure 7 As shown, Figure 7 This is a schematic diagram of another image resolution adjustment device according to an embodiment of the present invention.

[0381] The device may include the following units.

[0382] The second acquisition unit 401 is used to acquire the image to be adjusted and the resolution of the second target; the resolution of the image to be adjusted is lower than the resolution of the second target.

[0383] The direction determination unit 402 is used to determine M pixel expansion directions based on the second target resolution and the resolution of the image to be adjusted; M is a positive integer and M≥1; wherein, the product of (M+1) and the resolution of the image to be adjusted is greater than or equal to the second target resolution;

[0384] The second feature unit 403 is used to acquire M edge feature information in the M pixel expansion directions for the image to be adjusted;

[0385] The second integration unit 404 is used to integrate the image to be adjusted and M edge feature information to obtain a target image with a resolution of the second target resolution.

[0386] Optionally, the second feature unit 403 is used for:

[0387] For the image to be adjusted, M edge feature images are extracted using M preset convolution kernels corresponding to M pixel expansion directions; the resolution of the edge feature images is the same as the resolution of the image to be adjusted.

[0388] Different pixel expansion directions correspond to different preset convolution kernels; the preset convolution kernels are used to extract the gradient feature information between pixels in the corresponding pixel expansion direction.

[0389] Optionally, the second integration unit 404 is used for:

[0390] The image to be adjusted and M edge feature images are input into a pre-trained image ensemble network to obtain a target image with a resolution of the second target resolution output by the image ensemble network; the target image and the image to be adjusted have the same number of channels.

[0391] Optionally, the image combination network is used for:

[0392] For each of the M edge feature images, channel-wise overlay is performed with the image to be adjusted to obtain M overlay results;

[0393] For the M overlay results and the image to be adjusted, features are extracted using the representation layer to obtain M+1 images to be combined with the same resolution and number of channels as the image to be adjusted.

[0394] Input M+1 images to be combined into the combination layer to obtain a target image with the second target resolution and the same number of channels as the images to be adjusted.

[0395] Optionally, the combination layer is used for:

[0396] Perform the following operations on each pixel in the image to be combined, corresponding to the image to be adjusted, to obtain the combined result:

[0397] Starting from the target pixel, expand the matrix into a pixel matrix containing M+1 pixels by M pixel expansion directions, and add the pixels in the other M images to be combined that are at the same position as the target pixel to the expanded pixel matrix.

[0398] Based on the obtained combination results, a target image with the second target resolution and the same number of channels as the image to be adjusted is obtained through preset image processing.

[0399] For a detailed explanation, please refer to the above method implementation examples.

[0400] This invention also provides a computer device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described method embodiments.

[0401] This invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the above method embodiments.

[0402] Figure 8 This is a schematic diagram of the hardware structure of a computer device configured according to an embodiment of the present invention. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, memory 1020, input / output interface 1030, and communication interface 1040 are internally connected to each other via the bus 1050.

[0403] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0404] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of the present invention are implemented by software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0405] The input / output interface 1030 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0406] The communication interface 1040 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0407] Bus 1050 includes a pathway for transmitting information between various components of the device, such as processor 1010, memory 1020, input / output interface 1030, and communication interface 1040.

[0408] It should be noted that although the above-described device only shows the processor 1010, memory 1020, input / output interface 1030, communication interface 1040, and bus 1050, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of the present invention, and not necessarily all the components shown in the figures.

[0409] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described method embodiments.

[0410] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the above-described method embodiments.

[0411] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0412] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the embodiments of the present invention, in essence or the part that contributes, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0413] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0414] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. In implementing the embodiments of the present invention, the functions of each module can be implemented in one or more software and / or hardware. Alternatively, some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0415] The above description is merely a specific implementation of the embodiments of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the embodiments of the present invention, and these improvements and modifications should also be considered as protection of the embodiments of the present invention.

[0416] In this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0417] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. The invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the following claims.

[0418] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. An image resolution adjustment method, characterized in that, include: Obtain the image to be adjusted and the initial target resolution; The resolution of the image to be adjusted is higher than the first target resolution; The image to be adjusted is divided into N candidate images with a resolution equal to the first target resolution; wherein, the information of each pixel in the image to be adjusted is contained in any candidate image; N is a positive integer and N>1; For the segmented candidate images, extract edge feature information; By combining the N candidate images obtained from the segmentation with the extracted edge feature information, a target image with a resolution equal to the first target resolution is obtained; The step of dividing the image to be adjusted into N candidate images with a resolution equal to the first target resolution includes: Based on the first target resolution, several pixel matrices of the same size are divided from the image to be adjusted; each pixel matrix includes N pixels. In the image to be adjusted, the number of single-row pixel matrices divided horizontally is the same as the number of single-row pixels of the first target resolution, and the number of single-column pixel matrices divided vertically is the same as the number of single-column pixels of the first target resolution. Based on the divided pixel matrix, N candidate images with a resolution equal to the first target resolution are extracted.

2. The method according to claim 1, characterized in that, Pixels at the same position in different candidate images belong to the same pixel matrix; different pixels in any candidate image are located at the same position in their respective pixel matrix.

3. The method according to claim 2, characterized in that, The step of extracting edge feature information from the segmented candidate images includes: Select any candidate image as the starting image; For the initial image, using N-1 preset convolutional kernels corresponding to N-1 other candidate images, N-1 edge feature images are extracted for each of the N-1 other candidate images; the resolution of the edge feature images is the first target resolution; Different candidate images correspond to different preset convolution kernels; the preset convolution kernels are used to extract gradient feature information between pixels in a preset direction; the preset direction is the relative direction of different pixels at the same position in the same pixel matrix between the corresponding candidate image and the starting image.

4. The method according to claim 3, characterized in that, The N candidate images obtained through comprehensive segmentation and the extracted edge feature information are combined to obtain a target image with a resolution equal to the first target resolution, including: The N candidate images obtained from the segmentation and the N-1 edge feature images extracted are input into a pre-trained image channel merging network to obtain a target image with a resolution of the first target resolution output by the image channel merging network.

5. The method according to claim 4, characterized in that, The image channel merging network is used for: For the N-1 other candidate images, each image is superimposed with its corresponding edge feature image to obtain N-1 superimposed images; For the N-1 superimposed images and the starting image, features are extracted using the representation layer to obtain N convolutional feature images; The N convolutional feature images are superimposed on each other by channels, and then the channel merging layer is used to extract features to obtain a target image with the same resolution as the first target resolution and the same number of channels as the image to be adjusted.

6. The method according to claim 1, characterized in that, The step of extracting edge feature information from the segmented candidate images includes: For each identical location among the N candidate images obtained from the segmentation, perform the following operations to obtain a max-pooling feature map with a resolution of the first target resolution, and determine the max-pooling feature map as the extracted edge feature information: The pixel value with the largest value among the pixel values ​​of the N pixels at the target position in the N candidate images is determined as the pixel value of the pixel at the target position in the max pooling feature map.

7. The method according to claim 6, characterized in that, The N candidate images obtained through comprehensive segmentation and the extracted edge feature information are combined to obtain a target image with a resolution equal to the first target resolution, including: The N candidate images obtained from the segmentation and the max pooling feature map are input into a pre-trained image channel fusion network to obtain a target image with a resolution of the first target resolution output by the image channel fusion network.

8. The method according to claim 7, characterized in that, The image channel fusion network is used for: For the N candidate images, each image is superimposed with the max pooling feature map through channels to obtain N superimposed results; For the N superimposed results, features are extracted using the representation layer to obtain N alternative images with the same resolution as the first target resolution and the same number of channels as the image to be adjusted. The N candidate images are input into a max pooling layer to obtain a target image with the same resolution as the first target resolution and the same number of channels as the image to be adjusted.

9. An image resolution adjustment device, characterized in that, include: The first acquisition unit is used to acquire the image to be adjusted and the first target resolution; The resolution of the image to be adjusted is higher than the first target resolution; A segmentation unit is used to divide the image to be adjusted into N candidate images with a resolution of the first target resolution; wherein, the information of each pixel in the image to be adjusted is contained in any candidate image; N is a positive integer and N>1; The first feature unit is used to extract edge feature information from the segmented candidate image; The first integration unit is used to integrate the N candidate images obtained from the division and the extracted edge feature information to obtain a target image with a resolution of the first target resolution; The partitioning unit is further used for: Based on the first target resolution, several pixel matrices of the same size are divided from the image to be adjusted; each pixel matrix includes N pixels. In the image to be adjusted, the number of single-row pixel matrices divided horizontally is the same as the number of single-row pixels of the first target resolution, and the number of single-column pixel matrices divided vertically is the same as the number of single-column pixels of the first target resolution. Based on the divided pixel matrix, N candidate images with a resolution equal to the first target resolution are extracted.

10. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any one of claims 1 to 8.

Images