Image super-division method and device, equipment and storage medium
Patent Information
- Application Number
- CN202380008026.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-03-06
- Publication Date
- 2024-11-08
AI Technical Summary
In the existing technology, the computing cost required for equipment to perform image super-resolution is relatively high. In particular, it is difficult to achieve real-time image super-resolution on devices with less computing resources, which affects the clarity of real-time transmission services such as video conferencing.
By introducing a preset channel attention module into the image super-resolution network, the result weights are filtered based on the initial weights, reducing the number of weights involved in calculations, reducing computing resource requirements, and saving computing costs.
Without affecting the image super-resolution effect, the consumption of computing resources is reduced, the computing cost is reduced, and the efficiency of image super-resolution and the processing capability of the device are improved.
Smart Images

Figure CN118922853A_ABST
Abstract
Description
Image super-resolution method, device, equipment and storage medium Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image super-resolution method, device, equipment and storage medium. Background Art
[0002] Currently, many services require equipment to perform image super-resolution. For example, during a video conference, the transmitting device typically reduces the resolution of captured video frames for network transmission. The receiving device then super-resolutions the received low-resolution video frames to increase the resolution and thus improve the clarity of the video conference.
[0003] However, the computational cost required for devices to perform image super-resolution is often high.
[0004] Summary of the Invention
[0005] The present invention provides an image super-resolution method, apparatus, device and storage medium to address the deficiencies in related technologies.
[0006] According to a first aspect of an embodiment of the present invention, there is provided an image super-resolution method, comprising:
[0007] Determining a super-resolution requirement; the super-resolution requirement includes: converting an image of a first resolution into an image of a second resolution, the second resolution being greater than the first resolution;
[0008] Determining a preset image super-resolution network that meets the super-resolution requirement; inputting the image to be super-resolved at a first resolution into the preset image super-resolution network, and obtaining a result image of a second resolution output by the preset image super-resolution network;
[0009] The preset image super-resolution network is used to: extract image features based on a preset channel attention module;
[0010] The preset channel attention module is used to: determine the initial weight corresponding to the channel image according to the input feature map; determine the initial weight that meets the preset weight condition as the result weight; and output the result feature map according to the determined result weight.
[0011] Optionally, the minimum absolute value of the determined result weights is greater than or equal to the maximum absolute value of the initial weights that do not meet the preset weight condition.
[0012] Optionally, the preset weight condition includes at least one of the following:
[0013] The absolute value of the initial weight is greater than the preset weight lower limit;
[0014] When the initial weights are sorted in descending order according to the absolute value, the sorting sequence number is smaller than the preset sequence number;
[0015] When the initial weights are sorted in descending order according to absolute value, the sorting sequence number is in the top N%.
[0016] Optionally, outputting a result feature map according to the determined result weight includes:
[0017] Calculate the product of the determined result weight and the corresponding channel image respectively to obtain the corresponding single-channel feature map;
[0018] The resulting single-channel feature maps are stacked to obtain the resulting feature map.
[0019] Optionally, determining the initial weight corresponding to the channel image according to the input feature map includes at least one of the following:
[0020] Extract features from the input feature map and obtain the initial weights corresponding to the channel image;
[0021] Perform pooling processing on the input feature map to obtain a pooling result; extract features from the obtained pooling result to obtain the initial weight corresponding to the channel image;
[0022] Extract features from the input feature map; perform pooling on the extracted features to obtain a pooling result; extract features from the obtained pooling result to obtain the initial weights corresponding to the channel image.
[0023] Optionally, the pooling process includes at least one of the following:
[0024] Perform global maximum pooling processing;
[0025] Perform global average pooling processing;
[0026] Perform global maximum pooling and global average pooling respectively to obtain two pooling features, and then stack the two pooling features.
[0027] Optionally, determining a preset image super-resolution network that meets the super-resolution requirement includes:
[0028] Determining an original image super-resolution network; the original image super-resolution network is used to extract image features from an input image of a first resolution, and output a result image of a second resolution based on the extracted image features;
[0029] The original image super-resolution network is processed by a preset process to obtain a preset image super-resolution network that meets the super-resolution requirements.
[0030] Optionally, the original image super-resolution network is specifically used to: extract image features based on the original channel attention module;
[0031] The original channel attention module is used to: determine the initial weight corresponding to the channel image according to the input feature map; output the original feature map according to the determined initial weight;
[0032] The preset processing includes: according to the super-resolution requirement, setting preset weight conditions for one or more original channel attention modules in the original image super-resolution network, and converting them into preset channel attention modules, so that the preset image super-resolution network obtained after the conversion meets the super-resolution requirement.
[0033] Optionally, the super-resolution requirement also includes a computing power limitation requirement;
[0034] Determining the original image super-resolution network includes:
[0035] Determine the upper limit of the number of original channel modules according to the computing power limitation requirement in the super-resolution requirement, and determine the original image super-resolution network that meets the super-resolution requirement;
[0036] The number of original channel attention modules included in the original image super-resolution network is less than or equal to the upper limit of the number of original channel modules.
[0037] Optionally, the super-resolution requirement also includes a computing power limitation requirement;
[0038] The step of setting a preset weight condition for one or more original channel attention modules in the original image super-resolution network and converting them into preset channel attention modules includes:
[0039] Determine the number M of channel modules that need to be converted based on the computing power limitation requirements in the super-resolution requirements;
[0040] For the M original channel attention modules in the original image super-resolution network, preset weight conditions are set and converted into M preset channel attention modules.
[0041] Optionally, the super-resolution requirement also includes a computing power limitation requirement;
[0042] The step of setting preset weight conditions for one or more original channel attention modules in the original image super-resolution network and converting them into preset channel attention modules includes:
[0043] According to the computing power limitation requirements in the super-resolution requirements, determine the number M of original channel attention modules that need to be converted and the channel image retention ratio corresponding to each original channel attention module that needs to be converted;
[0044] For the M original channel attention modules in the original image super-resolution network, preset weight conditions are set according to the corresponding channel image retention ratios, and converted into M preset channel attention modules;
[0045] The preset weight condition includes: when the initial weights are sorted in descending order according to absolute value, the sorting sequence number is in the top N%; the N% is the corresponding channel image retention ratio.
[0046] Optionally, the super-resolution requirement also includes a computing power limitation requirement;
[0047] The determining of a preset image super-resolution network that meets the super-resolution requirement includes:
[0048] Determining an upper limit on the number of preset channel modules according to the computing power limitation requirement in the super-resolution requirement, and determining a preset image super-resolution network that meets the super-resolution requirement;
[0049] The number of preset channel attention modules included in the preset image super-resolution network is less than or equal to the upper limit of the number of preset channel modules.
[0050] Optionally, the super-resolution requirement also includes a computing power limitation requirement;
[0051] The determining of a preset image super-resolution network that meets the super-resolution requirement includes:
[0052] Determine, based on the computing power limitation requirement in the super-resolution requirement, a channel image retention ratio corresponding to a preset channel attention module; and determine a preset image super-resolution network that meets the super-resolution requirement;
[0053] In the preset image super-resolution network, the preset weight conditions in the preset channel attention module include: when the initial weights are sorted in order from large to small, the sorting sequence number is in the top N%; the N% is the corresponding channel image retention ratio.
[0054] Optionally, the preset image super-resolution network in the training process includes an original convolutional layer;
[0055] The original convolution layer is used to extract convolution features from the input feature map through at least two branches, obtain at least two branch convolution feature maps, and output the sum of the obtained branch convolution feature maps.
[0056] Optionally, at least one branch in the original convolution layer includes: a directional convolution layer for extracting inter-pixel gradient features in a preset direction.
[0057] Optionally, the preset image super-resolution network includes: a preset convolutional layer;
[0058] The preset convolutional layer is obtained by merging the trained preset image super-resolution network;
[0059] The merging process includes: performing merging calculation on at least two branches of the original convolutional layer in the trained preset image super-resolution network to obtain a corresponding preset convolutional layer;
[0060] The preset convolution layer is used to perform a single convolution operation on the input feature map, and output the sum of at least two branch result feature maps in the corresponding original convolution layer.
[0061] Optionally, the input of at least one preset channel attention module includes the output of a preset convolutional layer.
[0062] Optionally, extracting image features from the input image of the first resolution and outputting a result image of the second resolution based on the extracted image features includes:
[0063] Image features are extracted from an input image of a first resolution, and a first intermediate image of a second resolution is obtained based on the extracted image features. A second intermediate image of a second resolution is calculated using a preset calculation method for the input image of the first resolution. A result image of the second resolution is output based on the sum of the first intermediate image and the second intermediate image.
[0064] Optionally, the method is applied to a computing device, and the preset image super-resolution network is deployed in the computing device.
[0065] According to a second aspect of an embodiment of the present invention, there is provided an image super-resolution apparatus, comprising:
[0066] A requirement unit, configured to determine a super-resolution requirement; the super-resolution requirement includes: converting an image of a first resolution into an image of a second resolution, the second resolution being greater than the first resolution;
[0067] A network unit, configured to determine a preset image super-resolution network that meets the super-resolution requirement;
[0068] A result unit, configured to input the to-be-super-resolved image of the first resolution into the preset image super-resolution network, and obtain a result image of the second resolution output by the preset image super-resolution network;
[0069] The preset image super-resolution network is used to: extract image features based on a preset channel attention module;
[0070] The preset channel attention module is used to: determine the initial weight corresponding to the channel image according to the input feature map; determine the initial weight that meets the preset weight condition as the result weight; and output the result feature map according to the determined result weight.
[0071] Optionally, the minimum absolute value of the determined result weights is greater than or equal to the maximum absolute value of the initial weights that do not meet the preset weight condition.
[0072] Optionally, the preset weight condition includes at least one of the following:
[0073] The absolute value of the initial weight is greater than the preset weight lower limit;
[0074] When the initial weights are sorted in descending order according to the absolute value, the sorting sequence number is smaller than the preset sequence number;
[0075] When the initial weights are sorted in descending order according to absolute value, the sorting sequence number is in the top N%.
[0076] Optionally, the preset channel attention module is used to:
[0077] Calculate the product of the determined result weight and the corresponding channel image respectively to obtain the corresponding single-channel feature map;
[0078] The resulting single-channel feature maps are stacked to obtain the resulting feature map.
[0079] Optionally, the preset channel attention module is used to determine the initial weight corresponding to the channel image by at least one of the following methods:
[0080] Extract features from the input feature map and obtain the initial weights corresponding to the channel image;
[0081] Perform pooling processing on the input feature map to obtain a pooling result; extract features from the obtained pooling result to obtain the initial weight corresponding to the channel image;
[0082] Extract features from the input feature map; perform pooling on the extracted features to obtain a pooling result; extract features from the obtained pooling result to obtain the initial weights corresponding to the channel image.
[0083] Optionally, the pooling process includes at least one of the following:
[0084] Perform global maximum pooling processing;
[0085] Perform global average pooling processing;
[0086] Perform global maximum pooling and global average pooling respectively to obtain two pooling features, and then stack the two pooling features.
[0087] Optionally, the network unit is configured to:
[0088] Determining an original image super-resolution network; the original image super-resolution network is used to extract image features from an input image of a first resolution, and output a result image of a second resolution based on the extracted image features;
[0089] The original image super-resolution network is processed by a preset process to obtain a preset image super-resolution network that meets the super-resolution requirements.
[0090] Optionally, the original image super-resolution network is specifically used to: extract image features based on the original channel attention module;
[0091] The original channel attention module is used to: determine the initial weight corresponding to the channel image according to the input feature map; output the original feature map according to the determined initial weight;
[0092] The preset processing includes: according to the super-resolution requirement, setting preset weight conditions for one or more original channel attention modules in the original image super-resolution network, and converting them into preset channel attention modules, so that the preset image super-resolution network obtained after the conversion meets the super-resolution requirement.
[0093] Optionally, the super-resolution requirement also includes a computing power limitation requirement;
[0094] The network unit is used for:
[0095] Determine the upper limit of the number of original channel modules according to the computing power limitation requirement in the super-resolution requirement, and determine the original image super-resolution network that meets the super-resolution requirement;
[0096] The number of original channel attention modules included in the original image super-resolution network is less than or equal to the upper limit of the number of original channel modules.
[0097] Optionally, the super-resolution requirement also includes a computing power limitation requirement;
[0098] The preset processing includes:
[0099] Determine the number M of channel modules that need to be converted based on the computing power limitation requirements in the super-resolution requirements;
[0100] For the M original channel attention modules in the original image super-resolution network, preset weight conditions are set and converted into M preset channel attention modules.
[0101] Optionally, the super-resolution requirement also includes a computing power limitation requirement;
[0102] The preset processing includes:
[0103] According to the computing power limitation requirements in the super-resolution requirements, determine the number M of original channel attention modules that need to be converted and the channel image retention ratio corresponding to each original channel attention module that needs to be converted;
[0104] For the M original channel attention modules in the original image super-resolution network, preset weight conditions are set according to the corresponding channel image retention ratios, and converted into M preset channel attention modules;
[0105] The preset weight condition includes: when the initial weights are sorted in descending order according to absolute value, the sorting sequence number is in the top N%; the N% is the corresponding channel image retention ratio.
[0106] Optionally, the super-resolution requirement also includes a computing power limitation requirement;
[0107] The network unit is configured to:
[0108] Determining an upper limit on the number of preset channel modules according to the computing power limitation requirement in the super-resolution requirement, and determining a preset image super-resolution network that meets the super-resolution requirement;
[0109] The number of preset channel attention modules included in the preset image super-resolution network is less than or equal to the upper limit of the number of preset channel modules.
[0110] Optionally, the super-resolution requirement also includes a computing power limitation requirement;
[0111] The network unit is configured to:
[0112] Determine, based on the computing power limitation requirement in the super-resolution requirement, a channel image retention ratio corresponding to a preset channel attention module; and determine a preset image super-resolution network that meets the super-resolution requirement;
[0113] In the preset image super-resolution network, the preset weight conditions in the preset channel attention module include: when the initial weights are sorted in order from large to small, the sorting sequence number is in the top N%; the N% is the corresponding channel image retention ratio.
[0114] Optionally, the preset image super-resolution network in the training process includes an original convolutional layer;
[0115] The original convolution layer is used to extract convolution features from the input feature map through at least two branches, obtain at least two branch convolution feature maps, and output the sum of the obtained branch convolution feature maps.
[0116] Optionally, at least one branch in the original convolution layer includes: a directional convolution layer for extracting inter-pixel gradient features in a preset direction.
[0117] Optionally, the preset image super-resolution network includes: a preset convolutional layer;
[0118] The preset convolutional layer is obtained by merging the trained preset image super-resolution network;
[0119] The merging process includes: performing merging calculation on at least two branches of the original convolutional layer in the trained preset image super-resolution network to obtain a corresponding preset convolutional layer;
[0120] The preset convolution layer is used to perform a single convolution operation on the input feature map, and output the sum of at least two branch result feature maps in the corresponding original convolution layer.
[0121] Optionally, the input of at least one preset channel attention module includes the output of a preset convolutional layer.
[0122] Optionally, the preset image super-resolution network is used to:
[0123] Image features are extracted from an input image of a first resolution, and a first intermediate image of a second resolution is obtained based on the extracted image features. A second intermediate image of a second resolution is calculated using a preset calculation method for the input image of the first resolution. A result image of the second resolution is output based on the sum of the first intermediate image and the second intermediate image.
[0124] Optionally, the apparatus is applied to a computing device, and the preset image super-resolution network is deployed in the computing device.
[0125] According to the above embodiments, for the channel attention module in the preset image super-resolution network, the result weights are screened out through preset weight conditions and used to calculate the result feature map, thereby saving computing resources and reducing computing costs by reducing the number of weights involved in the calculation.
[0126] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0127] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0128] FIG1 is a schematic flow chart of an image super-resolution method according to an embodiment of the present invention;
[0129] FIG2 is a schematic diagram of the structure of a preset channel attention module according to an embodiment of the present invention;
[0130] FIG3 is a schematic diagram showing the structure of a preset image super-resolution network according to an embodiment of the present invention;
[0131] FIG4 is a schematic diagram of a directional convolution kernel according to an embodiment of the present invention;
[0132] FIG5 is a schematic diagram of a video conference according to an embodiment of the present invention;
[0133] FIG6 is a schematic structural diagram of an image super-resolution device according to an embodiment of the present invention;
[0134] FIG7 is a schematic diagram of the hardware structure of a computer device configured with the method according to an embodiment of the present invention. DETAILED DESCRIPTION
[0135] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.
[0136] Currently, many services require equipment to perform image super-resolution. For example, during a video conference, the transmitting device typically reduces the resolution of captured video frames for network transmission. The receiving device then super-resolutions the received low-resolution video frames to increase the resolution and thus improve the clarity of the video conference.
[0137] However, the computational cost required for devices to perform image super-resolution is often high.
[0138] Since image super-resolution on a device usually requires a high computational cost, it is often difficult for devices with less computing resources to perform image super-resolution locally.
[0139] Moreover, for some services with a strong real-time nature (e.g., video conferencing, live broadcasting, and other real-time transmission services), the computational cost required for real-time image super-resolution is higher, making it difficult to perform image super-resolution locally on devices with less computing resources (e.g., user terminals such as mobile phones and laptops).
[0140] In a specific example, when a user uses a mobile phone to start a video conference, the mobile phone often needs to consume a large computing cost to achieve real-time image super-resolution and improve the clarity of the video conference.
[0141] In order to address the deficiencies in the above-mentioned related technologies, an embodiment of the present invention provides an image super-resolution method.
[0142] In this method, it is first discovered that the channel attention mechanism can be used for image super-resolution. Specifically, a channel attention module for implementing the channel attention mechanism can be set in the image super-resolution network, and then the channel attention mechanism can be used to extract image features for image super-resolution.
[0143] Furthermore, in the channel attention mechanism, there is a step of determining weights for the channel images of the feature map, because the determined weights can reflect the importance of the channel images to a certain extent.
[0144] In order to reduce computational costs and save computing resources, the number of channel images in the channel attention mechanism can be reduced. Specifically, some less important channel images can be discarded according to the determined weights and not participate in subsequent calculations, thereby saving computing resources without affecting the image super-resolution effect as much as possible.
[0145] Therefore, this method can reduce the number of channel images by targeting the channel attention module in the image super-resolution network, saving computing resources and reducing computing costs. Furthermore, based on the corresponding weights of the channel images determined by the channel attention module, less important channel images can be reduced, thereby saving computing resources and reducing computing costs while minimizing the impact on the image super-resolution effect.
[0146] An image super-resolution method according to an embodiment of the present invention will be explained in detail below with reference to the accompanying drawings.
[0147] As shown in FIG1 , FIG1 is a flow chart of an image super-resolution method according to an embodiment of the present invention.
[0148] The process of this method does not limit the specific execution entity.
[0149] Optionally, the method flow may be executed by any electronic device. Optionally, the method flow may be executed by a user terminal, user equipment, server or other electronic device.
[0150] The method flow may include the following steps.
[0151] S101: Determine the over-resolution requirement.
[0152] Optionally, the super-resolution requirement may include: converting an image with a first resolution into an image with a second resolution, where the second resolution is greater than the first resolution.
[0153] S102: Determine a preset image super-resolution network that meets super-resolution requirements.
[0154] S103: Inputting the image to be super-resolved at the first resolution into a preset image super-resolution network to obtain a result image at the second resolution output by the preset image super-resolution network.
[0155] Optionally, the preset image super-resolution network can be used to extract image features from an input image of a first resolution, and output a result image of a second resolution based on the extracted image features.
[0156] Optionally, extracting image features may include extracting image features based on a preset channel attention module. In other words, the preset image super-resolution network may be used to extract image features based on the preset channel attention module. During the process of extracting image features, the preset image super-resolution network may use the preset channel attention module to extract image features.
[0157] Optionally, the preset channel attention module can be used to: determine the initial weight corresponding to the channel image based on the input feature map; determine the initial weight that meets the preset weight condition as the result weight; and output the result feature map based on the determined result weight.
[0158] The process of this method does not limit the specific form and source of the image to be super-resolved.
[0159] Optionally, the image to be super-resolved may be a video frame, which is used to implement video super-resolution. Specifically, step S103 may be performed on one or more video frames in the video to respectively increase the resolution, thereby improving the clarity and quality of the video.
[0160] Optionally, the image to be super-resolved may also be a blurred image, and the image clarity can be improved through image super-resolution in S103.
[0161] Optionally, when a result image is obtained, the result image may be further output to a terminal.
[0162] Optionally, the above method process can be performed remotely, so that after the image super-resolution is completed, the resulting image can be output to a terminal. The terminal here can specifically include a terminal device or a terminal client.
[0163] For example, the server may perform image super-resolution and then output the resulting image to the user's terminal client or terminal device.
[0164] Optionally, the image super-resolution process in the above method flow can be performed locally on the execution subject, so that after the image super-resolution, the result image can be output to a local terminal client. The terminal here specifically includes a terminal client.
[0165] For example, after receiving a low-resolution image locally, the local processor can execute the above method flow to output the super-resolved high-resolution image to the local terminal client for display. Of course, the image can specifically be a video frame to achieve video quality enhancement.
[0166] This method can filter out the result weights for the channel attention module in the preset image super-resolution network through preset weight conditions and use them to calculate the result feature map, thereby saving computing resources and reducing computing costs by reducing the number of weights involved in the calculation.
[0167] In an optional embodiment, the above method flow can be applied to a computing device, and a preset image super-resolution network can be deployed in the computing device to facilitate the computing device to execute the above method flow.
[0168] This embodiment does not limit the specific computing device. Optionally, the computing device may include: a terminal, a server, a mobile phone, a computer, an all-in-one conference machine, etc.
[0169] In this embodiment, by deploying a preset image super-resolution network into a computing device, the image super-resolution method of the above-mentioned method flow is used to execute, thereby saving the computing resources of the computing device for image super-resolution, reducing the computing cost of the computing device for image super-resolution, and improving the efficiency of image super-resolution.
[0170] The above method flow is explained in detail below from different aspects.
[0171] 1. About the preset channel attention module.
[0172] 1. Initial weight.
[0173] In an optional embodiment, for the convenience of description, the weights determined by the preset channel attention module for the channel image are described as initial weights. This concept is intended to distinguish them from the result weights obtained through subsequent screening.
[0174] Optionally, the initial weight may represent an evaluation prediction of the importance of the channel image by a preset channel attention module.
[0175] Optionally, the preset channel attention module can determine the initial weights for one or more channel images in the input feature map; it can also determine the initial weights for some or all channel images in the input feature map.
[0176] Optionally, the preset channel attention module can determine the corresponding initial weights for the channel image in the input feature map, or it can further extract features from the input feature map first, and then determine the corresponding initial weights for the channel image in the extracted feature map.
[0177] Optionally, the channel image used to determine the initial weights can be directly obtained from the input feature map, or features can be first extracted from the input feature map and then obtained from the extracted features.
[0178] For ease of description, the features extracted from the input feature map are described as intermediate feature maps. Therefore, optionally, the preset channel attention module can be used to determine the initial weights corresponding to the channel images in the input feature map and / or the intermediate feature map.
[0179] Of course, the channel image can also be described as a single-channel feature image.
[0180] The embodiment of the present invention does not limit the specific method of determining the initial weights by the preset channel attention module.
[0181] Optionally, features can be extracted from the input feature map to predict the initial weights corresponding to the channel image.
[0182] Optionally, pooling processing can be performed on the input feature map first, and then features can be extracted from the pooling result to predict the initial weights corresponding to the channel image.
[0183] This embodiment does not limit the method of extracting features. Optionally, the initial weights corresponding to the channel images can be predicted based on the pooling results through a convolutional layer, a fully connected layer, or other neural networks.
[0184] Optionally, features may be extracted from the input feature map first, and then pooling may be performed on the extracted features. Features may be extracted from the pooled results to predict initial weights corresponding to the channel images.
[0185] Among them, this embodiment does not limit the method of extracting features. Optionally, features can be extracted through a convolutional layer, a fully connected layer or other neural networks for the input feature map or pooling results.
[0186] In summary of the above-mentioned various initial weight determination methods, in an optional embodiment, determining the initial weight corresponding to the channel image according to the input feature map may include at least one of the following:
[0187] 1) Extract features from the input feature map and obtain the initial weights corresponding to the channel image.
[0188] 2) Perform pooling processing on the input feature map to obtain a pooling result; extract features from the obtained pooling result to obtain the initial weight corresponding to the channel image.
[0189] 3) Extract features from the input feature map; perform pooling on the extracted features to obtain a pooling result; extract features from the obtained pooling result to obtain the initial weight corresponding to the channel image.
[0190] This embodiment can improve the flexibility of the initial weight determination method by using multiple initial weight determination methods, thereby facilitating adaptation to different needs. In addition, by introducing pooling processing, the amount of data required for processing can be reduced, thereby improving the efficiency of determining the initial weight.
[0191] Optionally, in the preset image super-resolution network, different preset channel attention modules may adopt different initial weight determination methods.
[0192] This embodiment does not limit the specific pooling processing method and the specific method of extracting features.
[0193] Optionally, the pooling process may include at least one of the following: performing global maximum pooling process; performing global average pooling process; performing global maximum pooling process and global average pooling process respectively to obtain two pooling features, and then stacking the two obtained pooling features.
[0194] Optionally, the global maximum pooling process may include taking the maximum value of elements in the channel image, and specifically, pooling the channel image to the maximum value of the included elements.
[0195] Specifically, performing global max pooling on a feature (specifically, an input feature map or a feature extracted from an input feature map) may include taking the maximum value of all elements of one or more channel images in the feature. Specifically, taking the maximum value of all elements of all or part of the channel images in the feature. The size of the channel image after pooling may be 1*1, including the maximum value of the channel image elements.
[0196] Optionally, the global average pooling process may include taking an average value of elements in the channel image, and specifically, pooling the channel image as an average value of the included elements.
[0197] Specifically, performing global average pooling on a feature (specifically, an input feature map or a feature extracted from an input feature map) may include taking the average of all elements in one or more channel images of the feature. Specifically, taking the average of all elements in all or part of the channel images of the feature. The channel image after pooling may have a size of 1*1 and include the average of the channel image elements.
[0198] Optionally, two pooling features are stacked, specifically, the channel images of the two pooling features are stacked to increase the number of channels.
[0199] This embodiment can introduce the maximum value and the average value of the elements in the channel image through stacking, thereby increasing the amount of data information used to predict and determine the initial weights and improving the accuracy of the initial weights.
[0200] Of course, the above pooling processing examples are only for illustrative purposes and are not intended to limit the scope of the embodiments of the present invention.
[0201] The above-mentioned pooling processing embodiment can reduce the amount of data through specific pooling processing operations, thereby reducing the computational cost of subsequent prediction and determination of initial weights, and improving computational efficiency.
[0202] Regarding the explanation of feature extraction in the process of determining the initial weights, this embodiment does not limit the specific method of extracting features.
[0203] In an optional embodiment, features may be extracted using a convolutional layer, a fully connected layer, or other neural network. Specifically, features may be extracted using a preset convolutional layer. The description of the preset convolutional layer is provided below.
[0204] Optionally, the process of determining the initial weights may include extracting features from the input feature map to obtain initial weights corresponding to the channel image.
[0205] Specifically, the feature extraction process can be performed on the input feature map through a fully connected layer (convolutional layer or other neural network), and the initial weights corresponding to the channel images in the input feature map output by the fully connected layer (convolutional layer or other neural network) can be obtained. Specifically, the initial weights corresponding to the channel images in the input feature map can be obtained.
[0206] Optionally, the process of determining the initial weights may include performing pooling processing on the input feature map to obtain a pooling result; extracting features from the obtained pooling result to obtain the initial weights corresponding to the channel image.
[0207] Specifically, the pooling result is input into a fully connected layer (convolutional layer or other neural network) to extract features, and the initial weights corresponding to the channel images in the input feature map output by the fully connected layer (convolutional layer or other neural network) are obtained.
[0208] Optionally, the process of determining the initial weights may include extracting features from the input feature map; performing pooling processing on the extracted features to obtain a pooling result; and extracting features from the obtained pooling result to obtain the initial weights corresponding to the channel image.
[0209] Specifically, the input feature map can be extracted through a fully connected layer (convolutional layer or other neural network) to obtain an intermediate feature map output by the fully connected layer (convolutional layer or other neural network). For the pooling result, the pooling result can be input into a fully connected layer (convolutional layer or other neural network) to extract features and obtain the initial weights corresponding to the channel images in the intermediate feature map output by the fully connected layer (convolutional layer or other neural network).
[0210] 2. Preset weight conditions.
[0211] After the initial weights are determined, the result weights that meet the preset weight conditions can be screened out from the initial weights according to the preset weight conditions.
[0212] The embodiment of the present invention does not limit the specific preset weight condition.
[0213] In an optional embodiment, in order to minimize the impact of reducing the number of channel images and reduce the impact on the image super-resolution effect, a preset weight condition can be set to select channel images with higher importance and filter out channel images with lower importance.
[0214] Among them, since the determined initial weights are negative numbers, in order to facilitate screening, screening can be performed based on the absolute values of the initial weights.
[0215] Optionally, this embodiment does not limit the specific preset weight condition, as long as the minimum absolute value of the determined result weight is greater than or equal to the maximum absolute value of the initial weights that do not meet the preset weight condition.
[0216] In other words, this embodiment can set a preset weight condition and select the initial weight with a larger absolute value as the result weight for subsequent calculations, while the initial weight with a smaller absolute value can be discarded and does not participate in subsequent calculations. In this way, the channel image corresponding to the initial weight with a smaller absolute value (that is, the channel image with less importance) does not participate in subsequent calculations, thereby reducing the calculation cost.
[0217] For ease of understanding, specific examples of preset weight conditions are given below.
[0218] Optionally, the preset weight condition may include at least one of the following: the absolute value of the initial weight is greater than the preset weight lower limit; when the initial weights are sorted in order from large to small according to the absolute value, the sorting number is less than the preset number; when the initial weights are sorted in order from large to small according to the absolute value, the sorting number is in the top N%.
[0219] There is no limitation on the specific lower limit of the preset weight, the preset sequence number and the N value, nor is there a limitation on the specific setting method of the lower limit of the preset weight, the preset sequence number and the N value.
[0220] Optionally, in the preset image super-resolution network, different preset channel attention modules may adopt different preset weight conditions.
[0221] Optionally, the preset weight conditions can be automatically set by the machine based on actual needs, or can be customized by the user. Specifically, the preset weight conditions can be set based on the computing power limitations of the device itself.
[0222] In an optional example, the amount of computing power that needs to be saved by the preset channel attention module can be determined based on the requirements of computing power limitations, and the proportion of the number of channel images that needs to be reduced can be further determined to obtain the proportion N% of the number of channel images that can be retained. Then, the preset weight conditions can be set, including: when the initial weights are sorted in order from large to small according to the absolute value, the sorting sequence number is in the top N%.
[0223] Of course, the above preset weight conditions are only used for illustrative purposes.
[0224] In this embodiment, by using the specifically defined preset weight conditions, the standard for the result weight can be set, thereby facilitating the efficiency of determining the result weight, and the saved computing resources and computing costs can be conveniently determined through the set standards. The standard for the result weight in the preset weight conditions can also be set based on computing resource limitations or the desired computing cost savings to meet the corresponding needs.
[0225] 3. Result weight.
[0226] In the embodiment of the present invention, in order to distinguish it from the initial weight and facilitate description, the initial weight that meets the preset weight condition is described as the result weight.
[0227] Optionally, since the preset weight condition may include one or more conditions, the initial weight that satisfies some or all of the preset weight conditions may be determined as the result weight, or the initial weight that satisfies one or more preset weight conditions may be determined as the result weight.
[0228] After determining the result weight, the embodiment of the present invention does not limit the method of outputting the result feature map according to the result weight.
[0229] Optionally, the result feature map can be obtained based on the product of the result weight and the corresponding channel image; the initial weight that does not meet the preset weight condition can be set to 0, and the result feature map can be further obtained based on the product of the result weight and the corresponding channel image, as well as the product of other initial weights and the corresponding channel image; or the product of the result weight and the corresponding channel image can be convolved and features extracted through a convolution layer to obtain the result feature map.
[0230] Optionally, outputting a result feature map according to the determined result weight may include: respectively calculating the product of the determined result weight and the corresponding channel image to obtain a corresponding single-channel feature map; and stacking the obtained single-channel feature maps to obtain a result feature map.
[0231] This embodiment can improve the efficiency of calculating the result feature map by directly calculating the product of the result weight and the corresponding channel image.
[0232] Moreover, since the determination of the result feature map does not involve the initial weights that do not meet the preset weight conditions and the corresponding channel images, the number of channel images in the result feature map can be made less than the total number of initial weights, thereby reducing the amount of data and calculation, improving the efficiency of image super-resolution, and reducing the computational cost.
[0233] Taking the initial weights as an example, determining the initial weights corresponding to the channel images in the input feature map, by screening out the result weights, the number of channel images in the result feature map can be made less than the number of channel images in the input feature map, thereby reducing the amount of data and calculation, improving the efficiency of image super-resolution, and reducing the calculation cost.
[0234] 4. The structure of the preset channel attention module.
[0235] The embodiment of the present invention does not limit the specific structure of the preset channel attention module.
[0236] It should be noted that, optionally, different preset channel attention modules may have the same or different structures, or different specific implementations. Specifically, they may have the same or different initial weight determination methods, the same or different preset weight conditions, and the same or different result feature map determination methods.
[0237] For ease of understanding, as shown in FIG2 , FIG2 is a schematic structural diagram of a preset channel attention module according to an embodiment of the present invention.
[0238] Figure 2 shows two different preset channel attention modules (modules 1-2).
[0239] Module 1 includes feature extraction layer, pooling layer, fully connected layer and output layer.
[0240] Module 2 includes pooling layer, fully connected layer and output layer.
[0241] For module 1, the input feature map can be input into the feature extraction layer to extract the intermediate feature map, and then the intermediate feature map can be input into the pooling layer. Through the global maximum pooling and global average pooling in the pooling layer, the first pooling feature and the second pooling feature are obtained respectively. Then, the first pooling feature and the second pooling feature are stacked and input into the fully connected layer to obtain the initial weight corresponding to the intermediate feature map channel image output by the fully connected layer.
[0242] Then the initial weights and the intermediate feature map are input into the output layer, the result weights are screened out using the preset weight conditions, and then the products of each result weight and the corresponding channel image in the intermediate feature map are stacked into a feature map, which is determined as the result feature map and output.
[0243] For module 2, the input feature map can be input into the pooling layer. Through the global maximum pooling and global average pooling in the pooling layer, the third pooling feature and the fourth pooling feature are obtained respectively. Then, the third pooling feature and the fourth pooling feature are stacked and input into the fully connected layer to obtain the initial weight corresponding to the input feature map channel image output by the fully connected layer.
[0244] Then the initial weights and input feature maps are input into the output layer, the result weights are screened out using the preset weight conditions, and then the products of each result weight and the corresponding channel image in the input feature map are stacked into a feature map, which is determined as the result feature map and output.
[0245] It should be noted that the structure of the above-mentioned preset channel attention module is only for illustrative purposes and cannot limit the scope of the embodiments of the present invention.
[0246] 2. About the preset image super-resolution network.
[0247] 1. Preset the structure of the image super-resolution network.
[0248] The embodiment of the present invention does not limit the specific structure of the preset image super-resolution network.
[0249] In an optional embodiment, as long as the preset image super-resolution network can meet the super-resolution requirements and includes a preset channel attention module, the number of channel images can be reduced when extracting image features.
[0250] The embodiments of the present invention do not specifically limit the modules included in the preset image super-resolution network.
[0251] Optionally, the preset image super-resolution network may include at least: a preset channel attention module and a super-resolution module for image super-resolution.
[0252] Optionally, the preset image super-resolution network may also include at least one of the following: a convolutional layer, a spatial attention module, a feature extraction module, an activation function module, an upsampling module, a fully connected layer, a truncation module for limiting the value range (such as the image pixel value range), etc.
[0253] The embodiment of the present invention does not limit the number, order, and combination of modules in the preset image super-resolution network.
[0254] For ease of understanding, in an optional embodiment, a preset image super-resolution network can be used to implement feature extraction and upsampling.
[0255] Optionally, the feature extraction step can be implemented through a convolutional layer, a preset attention module, a spatial attention module, a fully connected layer, and an activation function module.
[0256] Optionally, upsampling can be achieved through convolutional layers and upsampling modules, etc.
[0257] In addition, optionally, the preset image super-resolution network can also achieve image super-resolution in a variety of different ways, and finally integrate the image super-resolution results of each way to obtain the final output result.
[0258] Optionally, the preset image super-resolution network can be used to: extract image features for an input image of a first resolution, and obtain a first intermediate image of a second resolution based on the extracted image features; calculate a second intermediate image of a second resolution for the input image of the first resolution using a preset calculation method; and output a result image of the second resolution based on the sum of the first intermediate image and the second intermediate image.
[0259] This embodiment does not limit the specific preset calculation method.
[0260] Optionally, the preset calculation method may be any image super-resolution method that can super-resolve an input image of a first resolution into an image of a second resolution. The preset calculation method may specifically include at least one of the following super-resolution interpolation methods: nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc.
[0261] Optionally, based on the sum of the first intermediate image and the second intermediate image, a result image of the second resolution is output. Specifically, the method may be: determining the sum of the first intermediate image and the second intermediate image as the result image of the second resolution; or performing feature extraction on the sum of the first intermediate image and the second intermediate image to obtain a result image of the second resolution.
[0262] This embodiment can improve the super-resolution effect of the preset image super-resolution network, reduce the training difficulty and convergence difficulty, and improve the training efficiency by setting the structure of the residual network.
[0263] For ease of understanding, an embodiment of the present invention provides a structural example of a preset image super-resolution network.
[0264] As shown in FIG3 , FIG3 is a schematic structural diagram of a preset image super-resolution network according to an embodiment of the present invention.
[0265] The preset image super-resolution network may include a convolutional layer, a preset channel attention module, an activation function, a spatial attention module, an upsampling layer, a bilinear interpolation layer, and a truncation layer.
[0266] Among them, the preset image super-resolution network can include 2 branches.
[0267] 1) Input the input image (resolution is the first resolution) into the convolution layer to extract features, and then input the extracted features into the combination of three modules in series.
[0268] Each module combination can include 1 preset channel attention module, 1 activation function, 1 spatial attention module and 1 activation function in series.
[0269] The features output by the last group of modules are combined and input into the upsampling layer to obtain the first intermediate image (with the second resolution).
[0270] 2) Input the input image into the bilinear interpolation layer to obtain a second intermediate image (with a resolution of the second resolution).
[0271] Afterwards, the first intermediate image and the second intermediate image may be added together, and then the values exceeding the image pixel value range may be corrected through a truncation layer to obtain a final result image (with a resolution of the second resolution).
[0272] 2. Preset the training method of image super-resolution network.
[0273] The embodiment of the present invention does not limit the specific training method of the preset image super-resolution network.
[0274] In an optional embodiment, since the preset channel attention module in the preset image super-resolution network can set a preset weight condition, it is not limited to setting the preset weight condition when training the preset image super-resolution network.
[0275] Optionally, when specifically training the preset image super-resolution network, the preset weight conditions may be directly set during the training process for training.
[0276] Optionally, since the preset weight conditions are primarily used to reduce the amount of data and, in some embodiments, can vary as needs change, setting the preset weight conditions during training may affect the training effect. Therefore, in order to improve the training effect of the preset image super-resolution network, the preset weight conditions may not be set during training of the preset image super-resolution network to filter the result weights.
[0277] Correspondingly, after the training is completed, the preset weight conditions can be further set for the preset image super-resolution network that has completed the training, and the screening of the result weights can be set to reduce the calculation cost.
[0278] For the convenience of description, the original image super-resolution network can be set as the source of the preset image super-resolution network.
[0279] Optionally, the original image super-resolution network can be used to extract image features from an input image of a first resolution, and output a result image of a second resolution based on the extracted image features.
[0280] Among them, the explanation of extracting image features can refer to the explanation of extracting features in the preset image super-resolution network above.
[0281] Optionally, the original image super-resolution network can be specifically used to: extract image features based on the original channel attention module; the original channel attention module can be used to: determine the initial weights corresponding to the channel image according to the input feature map; and output the original feature map according to the determined initial weights.
[0282] The explanation of the channel image and the initial weight can be found above.
[0283] The original channel attention module does not need to include preset weight conditions and does not filter the result weights.
[0284] The embodiment of the present invention does not limit the method of outputting the original feature map according to the initial weights.
[0285] Optionally, the original feature map can be obtained based on the product of the initial weight and the corresponding channel image; or the product of the initial weight and the corresponding channel image can be convolved and features extracted through a convolution layer to obtain the original feature map.
[0286] Optionally, outputting the original feature map according to the determined initial weights may include: respectively calculating the product of the determined initial weights and the corresponding channel images to obtain corresponding single-channel feature maps; and stacking the obtained single-channel feature maps to obtain the original feature map.
[0287] This embodiment can improve the efficiency of calculating the original feature map by directly calculating the product of the initial weight and the corresponding channel image.
[0288] Optionally, the sample set may be used to train the original image super-resolution network first, and then a preset weight condition may be set for the trained complete original image super-resolution network to implement the screening of the result weights and obtain the preset image super-resolution network.
[0289] This embodiment can flexibly set preset weight conditions for the original image super-resolution network to meet different needs.
[0290] 3. Method for determining the preset image super-resolution network.
[0291] Regarding S102 of determining a preset image super-resolution network that meets the super-resolution requirement, the embodiment of the present invention does not limit the specific method for determining the preset image super-resolution network.
[0292] In an optional embodiment, the executing entity may determine whether a preset image super-resolution network that meets the super-resolution requirements is deployed locally, or whether a preset image super-resolution network that can meet the super-resolution requirements by changing some parameters or structures is deployed locally, or whether a preset image super-resolution network that meets the super-resolution requirements can be obtained from a device other than the executing entity.
[0293] Optionally, determining a preset image super-resolution network that meets super-resolution requirements may include at least one of the following:
[0294] 1) Determine the original image super-resolution network that meets the super-resolution requirements and convert it into a preset image super-resolution network. This can be achieved by setting preset weight conditions.
[0295] More specifically, the original image super-resolution network that meets the super-resolution requirements can be obtained from a local device or a device outside the execution entity.
[0296] 2) Directly determine a preset image super-resolution network that meets the super-resolution requirements. Specifically, the preset image super-resolution network that meets the super-resolution requirements can be obtained from a local device or a device outside the execution entity.
[0297] 3) Determine an original image super-resolution network that meets some of the super-resolution requirements. For example, super-resolution requirements may include input and output resolution requirements, as well as computing power constraints. The original image super-resolution network that meets the input and output resolution requirements can be obtained locally or from a device external to the execution entity.
[0298] After that, the preset image super-resolution network can be obtained through conversion and setting to meet all the requirements of super-resolution.
[0299] 4) Determine a pre-defined image super-resolution network that meets some of the super-resolution requirements. For example, super-resolution requirements may include input and output resolution requirements, as well as computing power constraints. A pre-defined image super-resolution network that meets the input and output resolution requirements can be determined and obtained locally or from a device external to the execution entity.
[0300] Afterwards, the preset weight conditions can be updated to obtain a new preset image super-resolution network to meet all the requirements of the super-resolution requirements.
[0301] In an optional embodiment, determining a preset image super-resolution network that meets super-resolution requirements may include: determining an original image super-resolution network; and performing preset processing on the original image super-resolution network to obtain a preset image super-resolution network that meets super-resolution requirements.
[0302] This embodiment can obtain a preset image super-resolution network by using the original image super-resolution network and performing preset processing, thereby improving the flexibility of determining the preset image super-resolution network, facilitating corresponding preset processing according to super-resolution requirements, and obtaining a preset image super-resolution network that meets the requirements.
[0303] This embodiment does not limit the original image super-resolution network, as long as a preset image super-resolution network that meets the super-resolution requirements can be obtained through preset processing.
[0304] Optionally, the original image super-resolution network can be used to extract image features from an input image of a first resolution, and output a result image of a second resolution based on the extracted image features.
[0305] Optionally, the original image super-resolution network can be specifically used to extract image features based on the original channel attention module.
[0306] Optionally, the original channel attention module can be used to: determine the initial weight corresponding to the channel image according to the input feature map; and output the original feature map according to the determined initial weight.
[0307] Among them, the explanation of the original channel attention module can be found above.
[0308] This embodiment does not limit the specific method of determining the original image super-resolution network. Optionally, the original image super-resolution network that meets or does not meet the super-resolution requirement can be determined, specifically by obtaining the original image super-resolution network from a local device or a device outside the execution entity.
[0309] Optionally, an original image super-resolution network that meets the super-resolution requirements can be determined, and subsequent conversions can be performed based on actual needs, with preset weight conditions set to reduce computational costs.
[0310] Alternatively, it may be determined that the original image super-resolution network does not meet the super-resolution requirements. Specifically, it may not meet the computing power limit required by the super-resolution requirements. For example, the computing power required by the original image super-resolution network is too high, making it difficult for the execution entity to operate. Therefore, through subsequent conversion, preset weight conditions can be set to reduce computational costs and computing power requirements, thereby obtaining a preset image super-resolution network that meets the super-resolution requirements. For details, please refer to the following explanation.
[0311] This embodiment does not limit the specific method of the preset processing.
[0312] Optionally, the preset processing may include: according to the super-resolution requirements, setting preset weight conditions for one or more original channel attention modules in the original image super-resolution network, and converting them into preset channel attention modules, so that the preset image super-resolution network obtained after the conversion meets the super-resolution requirements.
[0313] Optionally, for the original image super-resolution network determined to meet the super-resolution requirements, the preset processing may include: setting preset weight conditions for one or more original channel attention modules in the original image super-resolution network, and converting them into preset channel attention modules. Specifically, the preset weight conditions may be set based on the desired computational cost savings.
[0314] Optionally, the preset processing may also include: modifying the execution logic in the original channel attention module so that it executes the corresponding logic of the preset channel attention module, specifically including a method of filtering out the result weights from the initial weights, and a method of outputting the result feature map according to the result weights.
[0315] The above embodiment can quickly convert the original image super-resolution network into the preset image super-resolution network through specific preset processing, thereby improving the efficiency and flexibility of determining the preset image super-resolution network.
[0316] Regarding S101 determining the super-resolution requirement, the embodiment of the present invention does not limit the specific form of the super-resolution requirement, nor does it limit the method for determining the super-resolution requirement.
[0317] Optionally, the super-resolution requirement may be determined based on a user request or device information.
[0318] In a specific example, the super-resolution requirement may be determined based on the computing resource allocation in the device information, or based on the image information required for super-resolution in the user request.
[0319] Optionally, the super-resolution requirement may also include a computing power limit requirement to control the computing cost of image super-resolution. The super-resolution requirement may also include other requirements, such as super-resolution real-time requirements and super-resolution efficiency requirements.
[0320] This embodiment does not limit the source and form of the computing power limitation requirement.
[0321] Optionally, the computing power limitation requirement can be determined based on the computing resources of the execution entity itself, or based on the computing resources in the execution entity that can be allocated to image super-resolution.
[0322] Optionally, the form of computing power limitation requirements may include at least one of the following: an upper limit on computing resources, an upper limit on the number of modules in the image super-resolution network, a number of channel images that need to be screened out, and the like.
[0323] It should be noted that the required computing power for the modules in the image super-resolution network can be determined through calculation. This can be achieved by reducing the number of modules or improving the criteria for channel image screening to meet the computing power constraints. The modules here can include the preset channel attention module, spatial attention module, convolutional layer, etc. in the image super-resolution network.
[0324] In an optional embodiment, the original image super-resolution network or the preset image super-resolution network may be determined according to the computing power limitation requirement included in the super-resolution requirement.
[0325] Optionally, determining the original image super-resolution network may include: determining an original image super-resolution network that meets super-resolution requirements.
[0326] Alternatively, an upper limit on the number of original channel modules that meets the computing power constraint can be directly determined. Specifically, determining the original image super-resolution network can include determining an upper limit on the number of original channel modules based on the computing power constraint in the super-resolution requirements, and determining an original image super-resolution network that meets the super-resolution requirements. The number of original channel attention modules included in the original image super-resolution network can be less than or equal to the upper limit on the number of original channel modules.
[0327] Among them, you can directly obtain the original image super-resolution network with the number of original channel attention modules less than or equal to the upper limit of the number of original channel modules, or you can delete some original channel attention modules from the original image super-resolution network until the computing power limit requirements are met.
[0328] This embodiment can meet the computing power limitation requirement by limiting the number of original channel attention modules, thereby improving the efficiency of determining the preset image super-resolution network.
[0329] Optionally, during the conversion, the conversion can be performed and preset weight conditions can be set according to the computing power limit requirements.
[0330] Optionally, for one or more original channel attention modules in the original image super-resolution network, preset weight conditions are set and converted into preset channel attention modules. Specifically, it can be: according to the computing power limitation requirements in the super-resolution requirements, the number M of channel modules that need to be converted is determined; for the M original channel attention modules in the original image super-resolution network, preset weight conditions are set and converted into M preset channel attention modules.
[0331] This embodiment can meet the computing power limitation requirement by limiting the number of original channel attention modules that need to be converted, thereby improving the efficiency of determining the preset image super-resolution network.
[0332] Optionally, for one or more original channel attention modules in the original image super-resolution network, preset weight conditions are set respectively and converted into preset channel attention modules. Specifically, it can be: according to the computing power limitation requirements in the super-resolution requirements, determine the number M of original channel attention modules that need to be converted and the channel image retention ratio corresponding to each original channel attention module that needs to be converted; for the M original channel attention modules in the original image super-resolution network, preset weight conditions are set respectively according to the corresponding channel image retention ratios, and converted into M preset channel attention modules; the preset weight conditions may include: when the initial weights are sorted in order from large to small according to the absolute value, the sorting sequence number is in the top N%; N% is the corresponding channel image retention ratio.
[0333] Among them, different original channel attention modules can be set to correspond to different channel image retention ratios.
[0334] This embodiment can meet the computing power limitation requirements by limiting the number of original channel attention modules that need to be converted and the number of channel images retained by the preset channel attention modules after conversion, thereby improving the efficiency of determining the preset image super-resolution network.
[0335] In another optional embodiment, a preset image super-resolution network that meets the super-resolution requirements may also be directly determined.
[0336] Optionally, determining a preset image super-resolution network that meets the super-resolution requirements can specifically include: determining an upper limit on the number of preset channel modules based on the computing power limitation requirements in the super-resolution requirements, and determining a preset image super-resolution network that meets the super-resolution requirements; the number of preset channel attention modules included in the preset image super-resolution network can be less than or equal to the upper limit on the number of preset channel modules.
[0337] Specifically, this can be achieved by deleting the preset channel attention module, or by directly obtaining a preset image super-resolution network in which the number of preset channel attention modules is less than or equal to the upper limit of the number of preset channel modules.
[0338] This embodiment can meet the computing power limitation requirement by limiting the number of preset channel attention modules, thereby improving the efficiency of determining the preset image super-resolution network.
[0339] Optionally, determining a preset image super-resolution network that meets the super-resolution requirements can specifically include: determining the channel image retention ratio corresponding to the preset channel attention module based on the computing power limitation requirements in the super-resolution requirements; determining a preset image super-resolution network that meets the super-resolution requirements; in the preset image super-resolution network, the preset weight conditions in the preset channel attention module include: when the initial weights are sorted in order from large to small, the sorting sequence number is in the top N%; N% is the corresponding channel image retention ratio.
[0340] Among them, different preset channel attention modules can be set to correspond to different channel image retention ratios.
[0341] Specifically, this can be achieved by deleting the preset channel attention module or by updating the preset weight conditions.
[0342] This embodiment can meet the computing power limitation requirement by limiting the number of preset channel attention modules and the number of channel images retained by the preset channel attention modules, thereby improving the efficiency of determining the preset image super-resolution network.
[0343] In addition, optionally, the number of other modules in the preset image super-resolution network can be reduced according to the computing power limitation requirements, for example, the number of spatial attention modules and / or convolutional layers can be reduced.
[0344] It should be noted that since the preset weight conditions can reduce channel images with low importance, the impact on the image super-resolution effect is controllable and small. According to the computing power limitation requirements, updating the preset weight conditions has little impact on the image super-resolution effect.
[0345] 3. About the preset image super-resolution network used to extract image features.
[0346] The embodiment of the present invention does not limit the specific method of extracting image features in the preset image super-resolution network.
[0347] Optionally, image features can be extracted by using convolutional layers, channel attention mechanisms, spatial attention mechanisms, activation functions, fully connected layers, etc.
[0348] In the process of extracting image features, the preset image super-resolution network can use at least one of the following to extract image features: a preset channel attention module, a spatial attention module, an activation function, a fully connected layer, a convolutional layer, etc.
[0349] The embodiment of the present invention does not limit the combination, quantity and order of these modules in the preset image super-resolution network.
[0350] In an optional embodiment, the preset image super-resolution network can extract image features through a convolutional layer.
[0351] This embodiment does not limit the form of the convolutional layer.
[0352] Optionally, the convolution layer may include multiple convolution branches, which extract image features from different angles and then synthesize them, so as to improve the training effect of the image super-resolution network.
[0353] Optionally, the convolution kernels of different convolution branches may be different, specifically including: different convolution kernel values, different convolution kernel sizes, etc. For example, a 2*2 convolution kernel and a 3*3 convolution kernel.
[0354] Optionally, the convolution branch can also be used to extract inter-pixel gradient features in a single direction, such as horizontal, vertical, or diagonal directions.
[0355] For ease of understanding, as shown in Figure 4, Figure 4 is a schematic diagram of a directional convolution kernel according to an embodiment of the present invention, which includes convolution kernels for extracting inter-pixel gradient features in eight directions.
[0356] Through different convolution branches, the comprehensiveness of feature extraction can be improved, the training effect of image super-resolution network can be improved, and the image super-resolution effect can be improved.
[0357] For multiple convolution branches, this embodiment can provide a way to reduce computational costs.
[0358] During the training process, the parameters can be updated in the form of convolution branches. However, after the training results, since the parameters of the convolution branches can be fixed, if the calculations are performed separately according to the form of convolution branches, the computational efficiency will be low.
[0359] Therefore, after the training is completed, different convolution branches can be merged, and the processing of each convolution branch can be achieved through a single convolution operation.
[0360] In order to facilitate the implementation of this merging operation, it is possible to limit the results of each convolution branch to be summed.
[0361] Since the convolution operation is additive, the results of each convolution branch are added together, so that the convolution kernels of each convolution branch can be combined to obtain a single convolution kernel for processing each convolution branch. Then, a single convolution operation can be performed on the input feature to process each convolution branch and obtain the sum of the results of each convolution branch.
[0362] For ease of understanding, in a specific example, the convolution layer may include two convolution branches, which are respectively expressed as: F1 = K1*X + B1; F2 = K2*X + B2.
[0363] Among them, F1 and F2 are the outputs of the two convolution branches respectively, K1 and K2 are the convolution kernels of the two convolution branches respectively, * represents the normal convolution operation, X is the same input of the two convolution branches, B1 and B2 are the offset coefficients of the two convolution branches respectively.
[0364] The overall output of the convolutional layer can be expressed as Ftotal, that is: Ftotal = F1 + F2 = K1*X + B1 + K2*X + B2 = (K1+K2)*X + (B1+B2)
[0365] Therefore, the two convolution branches in the convolution layer can be merged to obtain a single convolution operation with a convolution kernel of (K1+K2) and an offset coefficient of (B1+B2).
[0366] For other convolutional layer examples containing multiple convolution branches, it can be proved in a similar way that a single convolution operation can be obtained by merging the branches.
[0367] In another specific example, the convolution layer may include two convolution branches, one of which may adopt a conventional convolution operation: F3=K3*X+B3.
[0368] Another convolution branch can implement multiple convolutions. Taking two consecutive convolutions as an example: F4 = K4*(K4*X+B4)+B4.
[0369] The overall output of the convolutional layer can be expressed as Ftotal, that is: Ftotal = F3 + F4 = K3*X+B3+K4*(K4*X+B4)+B4 = K3*X+B3+K4*K4*X+K4*B4+B4 = (K3+K4*K4)*X+(B3+K4*B4+B4)
[0370] Therefore, the two convolution branches in the convolution layer can be merged to obtain a single convolution operation with a convolution kernel of (K3+K4*K4) and an offset coefficient of (B3+K4*B4+B4).
[0371] For other convolutional layer examples containing multiple convolution branches, it can be proved in a similar way that a single convolution operation can be obtained by merging the branches.
[0372] In another specific example, the convolution layer may include two convolution branches, one of which may adopt a conventional convolution operation: F5=K5*X+B5.
[0373] Another convolution branch can implement multiple convolutions. Taking two consecutive convolutions as an example: F6 = K6 (K6*X+B6) + B6.
[0374] in, Represents depthwise separable convolution operation (depthwise separable convolution, which can be understood as performing single-channel convolution on each channel and then merging them into multiple channels).
[0375] The overall output of the convolutional layer can be expressed as F, that is:
[0376]
[0377] Therefore, the two convolution branches in the convolution layer can be merged to obtain a single convolution operation with a convolution kernel of (K5+K6 K6) and an offset coefficient of (B5+K6 B6+B6).
[0378] For other convolutional layer examples containing multiple convolution branches, it can be proved in a similar way that a single convolution operation can be obtained by merging the branches.
[0379] Of course, in the above convolution operation, the output weights of different convolution branches can also be introduced, which does not affect the merging process and can also be converted into a single convolution operation.
[0380] Therefore, optionally, the preset image super-resolution network in the training process may include the original convolutional layer.
[0381] Optionally, the original convolution layer can be used to: extract convolution features from the input feature map through at least two branches respectively, obtain at least two branch convolution feature maps, and output the sum of the obtained branch convolution feature maps.
[0382] This embodiment does not limit the manner in which each branch extracts convolution features, and specifically, convolution features may be extracted by convolution.
[0383] This embodiment can facilitate subsequent merging processing and reduce computing costs by limiting the sum of branch results in the original convolutional layer.
[0384] Optionally, at least one branch in the original convolution layer includes: a directional convolution layer for extracting inter-pixel gradient features in a preset direction.
[0385] Optionally, different branches can use convolution kernels of different sizes, or use directional convolution layers that extract gradient features between pixels in different directions.
[0386] Among them, the original convolution layer can be in the form of containing various convolution branches, which is convenient for training.
[0387] In addition, the directional convolution layers included in different original convolution layers can extract inter-pixel gradient features in the same or different directions. This embodiment of the present invention is not limited thereto.
[0388] In a specific example, different original convolutional layers can be used alternately to extract inter-pixel gradient features in different directions.
[0389] For example, in the previous original convolution layer, directional convolution layers in the upward, upper right, right, and lower right directions can be used; and in the next original convolution layer, directional convolution layers in the downward, lower left, left, and upper left directions can be used, thereby improving the comprehensiveness of feature extraction and facilitating the improvement of the training effect of the image super-resolution network.
[0390] This embodiment can improve the training effect of the image super-resolution network and the effect of image super-resolution by extracting gradient features between pixels.
[0391] In an optional embodiment, after training is completed, the original convolutional layer can be merged so that the original output of the original convolutional layer can be obtained through a single convolution operation, thereby reducing the computational cost.
[0392] Optionally, the preset image super-resolution network may include: a preset convolutional layer.
[0393] Optionally, the preset convolutional layer may be obtained by merging a trained preset image super-resolution network.
[0394] Optionally, the merging process may include: performing merging calculation on at least two branches of the original convolutional layer in the trained preset image super-resolution network to obtain a corresponding preset convolutional layer.
[0395] Optionally, the preset convolution layer can be used to perform a single convolution operation on the input feature map, and output the sum of at least two branch result feature maps in the corresponding original convolution layer.
[0396] This embodiment can reduce the computational cost of the original convolutional layer and improve the efficiency of image super-resolution through merging processing.
[0397] In addition, optionally, the original image super-resolution network may include an original convolution layer, so that the original convolution layer can be trained during the training of the original image super-resolution network, and a merge process can be implemented after the training to obtain a preset convolution layer.
[0398] In an optional embodiment, the preset convolution layer can be used to extract image features. Therefore, the preset image super-resolution network can include the preset convolution layer.
[0399] In addition, since the preset channel attention module can also extract image features, optionally, the preset channel attention module can also include a preset convolution layer for extracting image features. Any step of extracting image features in the preset channel attention module can use the preset convolution layer to extract image features.
[0400] Specifically, in the preset channel attention module, the input feature map is first input into the preset convolutional layer to extract the intermediate feature map, and then the intermediate feature map is pooled and feature extracted.
[0401] Optionally, since the spatial attention module can also extract image features, the spatial attention module can also optionally include a preset convolution layer for extracting image features.
[0402] Optionally, the input of at least one preset channel attention module may include the output of a preset convolutional layer. The preset channel attention module may use the image features extracted by the preset convolutional layer as the input feature map.
[0403] Of course, other modules, such as the spatial attention module, can also use the image features extracted by the preset convolutional layer as input feature maps.
[0404] This embodiment can improve the effect of feature extraction and facilitate control of the required computing power by serially connecting a preset convolutional layer and a preset channel attention module.
[0405] 4. Application scenarios
[0406] For ease of understanding, the embodiment of the present invention further provides an application embodiment.
[0407] In a video conference scenario, as shown in FIG5 , FIG5 is a schematic diagram of a video conference according to an embodiment of the present invention.
[0408] Taking a two-person conference as an example, the display area 1 in Figure 5 is the area where the real-time stream of the current camera is displayed, and the captured content is the video conference screen of the person himself.
[0409] Display area 2 is a real-time stream transmitted over the network, and the content is the other party's video conference screen.
[0410] Because display area 1 is a local video stream, it can display the original resolution clarity of the current camera (such as 4K ultra-high-definition resolution), and the picture is clear.
[0411] However, due to the pressure of network transmission in multi-person conferences, in order to ensure real-time synchronization of video images, the other party's video stream is often compressed in resolution and bit rate, resulting in blurry and unclear images.
[0412] This embodiment mainly solves the unclear problem caused by the low resolution of the other party's video stream.
[0413] This embodiment can be applied to any other terminal / mobile terminal super-resolution scenarios, such as terminals (TV, all-in-one conference machine, electronic whiteboard, etc.) and mobile terminals (mobile phones, PADs, etc.).
[0414] In this embodiment, according to the super-resolution requirement, a preset image super-resolution network that meets the super-resolution requirement can be deployed on the terminal to achieve super-resolution of the other party's conference video image to improve the resolution and clarity.
[0415] Super-resolution requirements may include input and output resolution, as well as the computing power limitations of the terminal itself.
[0416] For the deployed terminals, a preset image super-resolution network that meets the super-resolution requirements can be optionally deployed on external devices. Since external devices often have more computing resources, a higher output resolution can be selected, or the super-resolution effect or the real-time performance of the super-resolution can be improved.
[0417] Specifically, for all-in-one conference machines, if costs permit and the super-resolution requirement is high, an external super-resolution box can be used and a preset image super-resolution network can be deployed on the super-resolution box.
[0418] Optionally, a preset image super-resolution network may also be deployed directly on the terminal.
[0419] Specifically, for all-in-one conference machines, if cost is a concern and super-resolution requirements are low, a pre-set image super-resolution network can be deployed on the board of the conference machine. This eliminates the need for additional equipment costs and allows for image super-resolution using the pre-set image super-resolution network, which can significantly improve image quality.
[0420] The specific computing power limitation requirements can be illustrated by the super-resolution scenario of an all-in-one conference machine.
[0421] In actual application, the number of modules can be adaptively set according to the computing power requirements of the terminal.
[0422] Assume that the computing power that the hardware of the all-in-one conference machine can provide to the super-resolution model is Mp (the specific unit is not limited here, for example, the unit can be Flops, MACS, Tops, etc., as long as the unit calculation of the algorithm remains consistent), and assume that in the preset image super-resolution network, the computing power required for all modules other than the preset channel attention module is Mo, the computing power required for a single preset channel attention module is Mc, and the number of preset channel attention modules is csN. Then, the corresponding number of preset channel attention modules that need to be retained, or the number of channel images that need to be retained in the preset channel attention module, can be calculated, so that the computing power of the overall preset image super-resolution network after update is less than or equal to Mp.
[0423] Corresponding to the above method embodiment, the embodiment of the present invention further provides an apparatus embodiment.
[0424] As shown in FIG6 , FIG6 is a structural diagram of an image super-resolution device according to an embodiment of the present invention.
[0425] The apparatus may include the following units.
[0426] The requirement unit 201 is configured to determine a super-resolution requirement. The super-resolution requirement includes converting an image of a first resolution into an image of a second resolution, where the second resolution is greater than the first resolution.
[0427] The network unit 202 is configured to determine a preset image super-resolution network that meets the super-resolution requirements.
[0428] The result unit 203 is configured to input the to-be-super-resolved image of the first resolution into a preset image super-resolution network, and obtain a result image of the second resolution output by the preset image super-resolution network.
[0429] Among them, the preset image super-resolution network can be used to extract image features based on a preset channel attention module.
[0430] The preset channel attention module can be used to: determine the initial weight corresponding to the channel image based on the input feature map; determine the initial weight that meets the preset weight condition as the result weight; and output the result feature map based on the determined result weight.
[0431] Optionally, the minimum absolute value of the determined result weights is greater than or equal to the maximum absolute value of the initial weights that do not meet the preset weight condition.
[0432] Optionally, the preset weight condition includes at least one of the following:
[0433] The absolute value of the initial weight is greater than the preset weight lower limit;
[0434] When the initial weights are sorted in descending order according to the absolute value, the sorting sequence number is smaller than the preset sequence number;
[0435] When the initial weights are sorted in descending order according to absolute value, the sorting sequence number is in the top N%.
[0436] Optionally, a preset channel attention module is used to:
[0437] Calculate the product of the determined result weight and the corresponding channel image respectively to obtain the corresponding single-channel feature map;
[0438] The resulting single-channel feature maps are stacked to obtain the resulting feature map.
[0439] Optionally, the preset channel attention module is used to determine the initial weight corresponding to the channel image by at least one of the following methods:
[0440] Extract features from the input feature map and obtain the initial weights corresponding to the channel image;
[0441] Perform pooling processing on the input feature map to obtain a pooling result; extract features from the obtained pooling result to obtain the initial weight corresponding to the channel image;
[0442] Extract features from the input feature map; perform pooling on the extracted features to obtain a pooling result; extract features from the obtained pooling result to obtain the initial weights corresponding to the channel image.
[0443] Optionally, the pooling process includes at least one of the following:
[0444] Perform global maximum pooling processing;
[0445] Perform global average pooling processing;
[0446] Perform global maximum pooling and global average pooling respectively to obtain two pooling features, and then stack the two pooling features.
[0447] Optionally, the network unit 202 is configured to:
[0448] Determining an original image super-resolution network; the original image super-resolution network is used to extract image features from an input image of a first resolution, and output a result image of a second resolution based on the extracted image features;
[0449] The original image super-resolution network is processed pre-set to obtain a preset image super-resolution network that meets the super-resolution requirements.
[0450] Optionally, the original image super-resolution network is specifically used to: extract image features based on the original channel attention module;
[0451] The original channel attention module is used to: determine the initial weight corresponding to the channel image according to the input feature map; output the original feature map according to the determined initial weight;
[0452] The preset processing includes: according to the super-resolution requirements, setting preset weight conditions for one or more original channel attention modules in the original image super-resolution network, and converting them into preset channel attention modules, so that the preset image super-resolution network obtained after the conversion meets the super-resolution requirements.
[0453] Optionally, the over-resolution requirement also includes a computing power limit requirement;
[0454] The network unit 202 is used for:
[0455] According to the computing power limit requirements in the super-resolution requirements, determine the upper limit of the number of original channel modules and determine the original image super-resolution network that meets the super-resolution requirements;
[0456] The number of original channel attention modules included in the original image super-resolution network is less than or equal to the upper limit of the number of original channel modules.
[0457] Optionally, the over-resolution requirement also includes a computing power limit requirement;
[0458] Preset processing, including:
[0459] Determine the number M of channel modules that need to be converted based on the computing power limit requirements in the super-resolution requirements;
[0460] For the M original channel attention modules in the original image super-resolution network, preset weight conditions are set and converted into M preset channel attention modules.
[0461] Optionally, the over-resolution requirement also includes a computing power limit requirement;
[0462] Preset processing, including:
[0463] According to the computing power limit requirements in the super-resolution requirements, determine the number M of original channel attention modules that need to be converted and the channel image retention ratio corresponding to each original channel attention module that needs to be converted;
[0464] For the M original channel attention modules in the original image super-resolution network, preset weight conditions are set according to the corresponding channel image retention ratios, and converted into M preset channel attention modules;
[0465] The preset weight conditions include: when the initial weights are sorted in descending order according to absolute values, the sorting sequence number is in the top N%; N% is the corresponding channel image retention ratio.
[0466] Optionally, the over-resolution requirement also includes a computing power limit requirement;
[0467] The network unit 202 is configured to:
[0468] According to the computing power limit requirements in the super-resolution requirements, determine the upper limit of the preset channel module number and determine the preset image super-resolution network that meets the super-resolution requirements;
[0469] The number of preset channel attention modules included in the preset image super-resolution network is less than or equal to the upper limit of the number of preset channel modules.
[0470] Optionally, the over-resolution requirement also includes a computing power limit requirement;
[0471] The network unit 202 is configured to:
[0472] According to the computing power limitation requirements in the super-resolution requirements, determine the channel image retention ratio corresponding to the preset channel attention module; determine the preset image super-resolution network that meets the super-resolution requirements;
[0473] In the preset image super-resolution network, the preset weight conditions in the preset channel attention module include: when the initial weights are sorted in order from large to small, the sorting sequence number is in the top N%; N% is the corresponding channel image retention ratio.
[0474] Optionally, the preset image super-resolution network in the training process includes an original convolutional layer;
[0475] The original convolution layer is used to extract convolution features from the input feature map through at least two branches, obtain at least two branch convolution feature maps, and output the sum of the obtained branch convolution feature maps.
[0476] Optionally, at least one branch in the original convolution layer includes: a directional convolution layer for extracting inter-pixel gradient features in a preset direction.
[0477] Optionally, the preset image super-resolution network includes: a preset convolutional layer;
[0478] The preset convolution layer is obtained by merging the trained preset image super-resolution network;
[0479] The merging process includes: merging at least two branches of the original convolution layer in the trained preset image super-resolution network to obtain a corresponding preset convolution layer;
[0480] The preset convolution layer is used to perform a single convolution operation on the input feature map and output the sum of the feature maps of at least two branches in the corresponding original convolution layer.
[0481] Optionally, the input of at least one preset channel attention module includes the output of a preset convolutional layer.
[0482] Optionally, the preset image super-resolution network is used to:
[0483] Image features are extracted from an input image of a first resolution, and a first intermediate image of a second resolution is obtained based on the extracted image features. A second intermediate image of a second resolution is calculated using a preset calculation method for the input image of the first resolution. A result image of the second resolution is output based on the sum of the first intermediate image and the second intermediate image.
[0484] Optionally, the above apparatus may be applied to a computing device, and the preset image super-resolution network may be deployed in the computing device.
[0485] For detailed explanation, please refer to the above method embodiment.
[0486] An embodiment of the present invention further provides a computer device, which includes at least a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the above method embodiments is implemented.
[0487] An embodiment of the present invention also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above-mentioned method embodiments.
[0488] FIG7 is a schematic diagram of the hardware structure of a computer device configured to implement the method according to an embodiment of the present invention, according to an embodiment of the present invention. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other within the device via the bus 1050.
[0489] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.
[0490] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present invention are implemented through software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.
[0491] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.
[0492] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).
[0493] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).
[0494] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, those skilled in the art will understand that the above device may only include the components necessary to implement the embodiments of the present invention, and does not necessarily include all the components shown in the figure.
[0495] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements any of the above method embodiments when executed by a processor.
[0496] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program implements any of the above method embodiments when executed by a processor.
[0497] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.
[0498] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the embodiments of the present invention can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the embodiments of the present invention can essentially or in other words, the contributing part can be embodied in the form of a software product. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.
[0499] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.
[0500] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the modules described as separate components may or may not be physically separated, and when implementing the embodiment of the present invention, the functions of each module can be implemented in the same one or more software and / or hardware. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment. A person of ordinary skill in the art can understand and implement it without paying any creative work.
[0501] The above description is only a specific implementation of the embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiment of the present invention. These improvements and modifications should also be regarded as protection for the embodiment of the present invention.
[0502] In the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance. The term "plurality" refers to two or more, unless otherwise clearly defined.
[0503] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the disclosure herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.
[0504] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. An image super-resolution method, characterized in that: include: Determine the oversubscription requirements; The super-resolution requirement includes: converting an image of a first resolution into an image of a second resolution, wherein the second resolution is greater than the first resolution; Determine a preset image super-resolution network that meets the super-resolution requirement; input the image to be super-resolution of the first resolution into the preset image super-resolution network, and obtain a result image of the second resolution output by the preset image super-resolution network; Wherein, the preset image super-resolution network is used to: extract image features based on a preset channel attention module; The preset channel attention module is used to: determine the initial weight corresponding to the channel image according to the input feature map; determine the initial weight that meets the preset weight condition as the result weight; and output the result feature map according to the determined result weight.
2. The method according to claim 1, characterized in that The minimum absolute value of the determined result weights is greater than or equal to the maximum absolute value of the initial weights that do not meet the preset weight condition.
3. The method according to claim 1, characterized in that: The preset weight condition includes at least one of the following: The absolute value of the initial weight is greater than the preset weight lower limit; When the initial weights are sorted in descending order according to the absolute value, the sorting sequence number is less than the preset sequence number; When the initial weights are sorted in descending order according to their absolute values, the sorting sequence number is in the first N%.
4. The method according to claim 1, characterized in that: The outputting the result feature map according to the determined result weight comprises: Calculate the product of the determined result weight and the corresponding channel image respectively to obtain the corresponding single channel feature map; The resulting single-channel feature maps are stacked to obtain the resulting feature map.
5. The method according to claim 1, characterized in that The step of determining the initial weight corresponding to the channel image according to the input feature map includes at least one of the following: Extract features from the input feature map and obtain the initial weights corresponding to the channel image; Perform pooling processing on the input feature map to obtain a pooling result; extract features from the obtained pooling result to obtain the initial weight corresponding to the channel image; Extract features from the input feature map; Pooling is performed on the extracted features to obtain a pooling result; features are extracted from the obtained pooling result to obtain initial weights corresponding to the channel image.
6. The method according to claim 5, characterized in that The pooling process includes at least one of the following: Perform global maximum pooling processing; Perform global average pooling processing; Perform global maximum pooling and global average pooling respectively to obtain two pooling features, and then The two pooling features are stacked.
7. The method according to claim 1, characterized in that The determining of a preset image super-resolution network that meets the super-resolution requirement includes: Determine an original image super-resolution network; the original image super-resolution network is used to extract image features from an input image of a first resolution, and output a result image of a second resolution according to the extracted image features; The original image super-resolution network is processed by a preset method to obtain a preset image super-resolution network that meets the super-resolution requirements.
8. The method according to claim 7, characterized in that The original image super-resolution network is specifically used to: extract image features based on the original channel attention module; The original channel attention module is used to: determine the initial weight corresponding to the channel image according to the input feature map; Outputting the original feature map according to the determined initial weights; The preset processing includes: according to the super-resolution requirement, setting preset weight conditions for one or more original channel attention modules in the original image super-resolution network, converting them into preset channel attention modules, so that the preset image super-resolution network obtained after the conversion meets the super-resolution requirement.
9. The method according to claim 8, characterized in that The over-resolution requirement also includes computing power limitation requirement; The determining of the original image super-resolution network comprises: According to the computing power limitation requirement in the super-resolution requirement, determining the upper limit of the number of original channel modules, and determining the original image super-resolution network that meets the super-resolution requirement; The number of original channel attention modules included in the original image super-resolution network is less than or equal to the upper limit of the number of original channel modules.
10. The method according to claim 8, characterized in that The over-resolution requirement also includes computing power limitation requirement; The step of setting a preset weight condition for one or more original channel attention modules in the original image super-resolution network and converting them into preset channel attention modules comprises: Determine the number M of channel modules that need to be converted according to the computing power limitation requirement in the super-resolution requirement; For the M original channel attention modules in the original image super-resolution network, preset weight conditions are set and converted into M preset channel attention modules.
11. The method according to claim 8, characterized in that The over-resolution requirement also includes computing power limitation requirement; The step of setting preset weight conditions for one or more original channel attention modules in the original image super-resolution network and converting them into preset channel attention modules comprises: According to the computing power limitation requirement in the super-resolution requirement, determine the number M of original channel attention modules that need to be converted and the channel image retention ratio corresponding to each original channel attention module that needs to be converted; For the M original channel attention modules in the original image super-resolution network, preset weight conditions are set according to the corresponding channel image retention ratios, and converted into M preset channel attention modules; The preset weight condition includes: when the initial weights are sorted in order from large to small according to the absolute value, the sorting sequence number is in the first N%; and the N% is the corresponding channel image retention ratio.
12. The method according to claim 1, characterized in that The over-resolution requirement also includes computing power limitation requirement; The determining of a preset image super-resolution network that meets the super-resolution requirement includes: Determine the upper limit of the number of preset channel modules according to the computing power limitation requirement in the super-resolution requirement, and determine a preset image super-resolution network that meets the super-resolution requirement; The number of preset channel attention modules included in the preset image super-resolution network is less than or equal to the upper limit of the number of preset channel modules.
13. The method according to claim 1, characterized in that The over-resolution requirement also includes computing power limitation requirement; The determining of a preset image super-resolution network that meets the super-resolution requirement includes: Determine the channel image retention ratio corresponding to the preset channel attention module according to the computing power limitation requirement in the super-resolution requirement; determine the preset image super-resolution network that meets the super-resolution requirement; In the preset image super-resolution network, the preset weight condition in the preset channel attention module includes: when the initial weights are sorted in order from large to small, the sorting sequence number is in the top N%; the N% is the corresponding channel image retention ratio.
14. The method according to claim 1, characterized in that The preset image super-resolution network during training includes the original convolutional layer; The original convolution layer is used to extract convolution features from the input feature map through at least two branches respectively, obtain at least two branch convolution feature maps, and output the sum of the obtained branch convolution feature maps.
15. The method according to claim 14, characterized in that At least one branch in the original convolution layer includes: a directional convolution layer for extracting inter-pixel gradient features in a preset direction.
16. The method according to claim 14 or 15, characterized in that The preset image super-resolution network includes: a preset convolutional layer; The preset convolutional layer is obtained by merging the trained preset image super-resolution network; The merging process includes: obtaining a corresponding preset convolution layer by merging and calculating at least two branches of the original convolution layer in the trained preset image super-resolution network; The preset convolution layer is used to perform a single convolution operation on the input feature map, and output the sum of at least two branch result feature maps in the corresponding original convolution layer.
17. The method according to claim 16, characterized in that The input of at least one preset channel attention module includes the output of a preset convolutional layer.
18. The method according to claim 1, characterized in that The preset image super-resolution network is used for: Extract image features from an input image of a first resolution, and obtain a first intermediate image of a second resolution based on the extracted image features; calculate a second intermediate image of a second resolution using a preset calculation method for the input image of the first resolution; and output a result image of a second resolution based on the sum of the first intermediate image and the second intermediate image.
19. The method according to claim 1, characterized in that The method is applied to a computing device, and the preset image super-resolution network is deployed in the computing device.
20. An image super-resolution device, characterized in that: include: Demand unit, used to determine the excess demand; The super-resolution requirement includes: converting an image of a first resolution into an image of a second resolution, wherein the second resolution is greater than the first resolution; A network unit, used to determine a preset image super-resolution network that meets the super-resolution requirement; A result unit, used for inputting the to-be-super-resolved image of the first resolution into the preset image super-resolution network, and obtaining a result image of the second resolution output by the preset image super-resolution network; Wherein, the preset image super-resolution network is used to: extract image features based on a preset channel attention module; The preset channel attention module is used to: determine the initial weight corresponding to the channel image according to the input feature map; determine the initial weight that meets the preset weight condition as the result weight; and output the result feature map according to the determined result weight.
21. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor so as to enable the at least one processor to perform the method according to any one of claims 1 to 19.
22. A computer-readable storage medium storing a computer program, characterized in that: The computer program implements the method according to any one of claims 1 to 19 when executed by a processor.
