Image super-resolution reconstruction method, device, equipment and storage medium

Abstract

Embodiments of the present application disclose a kind of image super-resolution reconstruction method, device, equipment and storage medium, related to image processing field, it includes: obtaining low-resolution image containing target object;Mask image of target object in low-resolution image is extracted;According to mask image, the low-resolution area image of target object is obtained in low-resolution image;Super-resolution neural network is handled low-resolution area image, and the super-resolution area image of target object is obtained;Low-resolution image is carried out up sampling operation, and the up sampling image is obtained;Fusion up sampling image and super-resolution area image, and the super-resolution image of low-resolution image is obtained, and up sampling image, super-resolution area image and super-resolution image have the same resolution.The technical problem that processing speed is slower in related art when carrying out super-resolution in video scene can be solved by using the above method.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image super-resolution reconstruction method, apparatus, device, and storage medium. Background Technology

[0002] Super-resolution (SR) reconstruction can be understood as reconstructing a corresponding high-resolution image based on a low-resolution image. SR has been widely used in video transmission scenarios such as video conferencing and video learning. For example, after the sending end collects video data, it transmits low-resolution video data over the network to save network resources and improve transmission speed. Then, the receiving end receives the video data and uses SR technology to reconstruct high-resolution video data to achieve high-definition video playback. Among some related technologies, the following two schemes are mainly used for SR of video data: one is to use a lightweight network to process the images in the video data frame by frame, performing SR on the displayed content in the images, thereby achieving SR of the video data. Examples include using lightweight networks such as FSRCNN, ESPCN, or IMDN for SR to process the images in the video data frame by frame; the other is to use networks such as FSRVR or EDVR for SR to process multiple frames of images in the video data, reconstructing a single high-resolution image based on multiple frames, thus achieving SR of the video data. In the process of realizing this invention, the inventors discovered the following defects in the above-mentioned related technologies: whether it is using a lightweight super-resolution network to process images in video data frame by frame, or reconstructing a high-resolution image from multiple frames of images in video data, both have slow processing speeds, which cannot meet the speed requirements of super-resolution in video scenarios and reduce the meeting experience of video conferencing. Summary of the Invention

[0003] One embodiment of this application provides an image super-resolution reconstruction method, apparatus, device, and storage medium to solve the technical problem of slow processing speed when performing super-resolution in video scenes in related technologies.

[0004] In a first aspect, embodiments of this application provide an image super-resolution reconstruction method, including:

[0005] Obtain a low-resolution image containing the target object;

[0006] Extract the mask image of the target object from the low-resolution image;

[0007] Based on the mask image, a low-resolution region image of the target object is obtained in the low-resolution image;

[0008] The low-resolution region image is processed using a super-resolution neural network to obtain a super-resolution region image of the target object;

[0009] The low-resolution image is upsampled to obtain an upsampled image;

[0010] The upsampled image and the super-resolution region image are fused to obtain the super-resolution image of the low-resolution image, wherein the upsampled image, the super-resolution region image, and the super-resolution image have the same resolution.

[0011] Secondly, embodiments of this application also provide an image super-resolution reconstruction apparatus, comprising:

[0012] The image acquisition module is used to acquire low-resolution images containing the target object.

[0013] A mask extraction module is used to extract the mask image of the target object in the low-resolution image;

[0014] A region determination module is used to obtain a low-resolution region image of the target object in the low-resolution image based on the mask image;

[0015] An image super-resolution module is used to process the low-resolution region image using a super-resolution neural network to obtain a super-resolution region image of the target object.

[0016] An image upsampling module is used to perform an upsampling operation on the low-resolution image to obtain an upsampled image;

[0017] An image fusion module is used to fuse the upsampled image and the super-resolution region image to obtain a super-resolution image of the low-resolution image, wherein the upsampled image, the super-resolution region image, and the super-resolution image have the same resolution.

[0018] Thirdly, embodiments of this application also provide an image super-resolution reconstruction device, comprising:

[0019] One or more processors;

[0020] Memory, used to store one or more programs;

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement the image super-resolution reconstruction method as described in the first aspect.

[0022] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the image super-resolution reconstruction method as described in the first aspect.

[0023] The aforementioned image super-resolution reconstruction method, apparatus, device, and storage medium, through acquiring a low-resolution image containing the target object, obtaining a mask image containing the target object based on the low-resolution image, then determining a low-resolution region image of the target object's location in the low-resolution image based on the mask image, using a super-resolution neural network to obtain a corresponding super-resolution region image, and performing an upsampling operation on the low-resolution image to obtain an upsampled image, and finally fusing the super-resolution region image and the upsampled image to obtain the final super-resolution image, solves the technical problem of slow processing speed when performing super-resolution in video scenes in related technologies. Compared to using a super-resolution neural network to perform super-resolution on all low-resolution images, using the target object's mask image can extract the image of the target object's location in the low-resolution image, and performing super-resolution only on the target object based on the super-resolution neural network can reduce the data processing volume of the super-resolution neural network, improve the processing speed and computational efficiency of the super-resolution neural network, and improve the resolution of non-target object areas by upsampling the low-resolution image, ensuring that the final super-resolution image not only contains a clearer target object but also includes non-target objects. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of data transmission in a video conferencing scenario;

[0025] Figure 2 A flowchart illustrating an image super-resolution reconstruction method provided in one embodiment of this application;

[0026] Figure 3 A flowchart illustrating another image super-resolution reconstruction method provided in one embodiment of this application;

[0027] Figure 4 A data flow diagram provided in one embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the structure of an image super-resolution reconstruction apparatus provided in one embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the structure of an image super-resolution reconstruction device provided in one embodiment of this application. Detailed Implementation

[0030] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the scope of the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present application are shown in the drawings, not the entire structure.

[0031] The image super-resolution reconstruction method provided in one embodiment of this application can be executed by an image super-resolution reconstruction device. This device can be implemented through software and / or hardware, and can consist of two or more physical entities, or a single physical entity. For example, the image super-resolution reconstruction device can be an electronic device with data processing and analysis capabilities, such as a computer, mobile phone, tablet, or interactive smart panel.

[0032] In one embodiment, the image super-resolution reconstruction device is used to perform super-resolution reconstruction, which can be applied in video communication scenarios such as video conferencing and video teaching, as well as other scenarios requiring super-resolution reconstruction. Taking a video conferencing scenario as an example, the data transmission diagram when applying super-resolution reconstruction to video conferencing is as follows. Figure 1 As shown. Figure 1 This is a schematic diagram of data transmission in a video conferencing scenario. (For reference) Figure 1 During a video conference, after a camera or webcam captures video data, the electronic devices in the conference first downsample the image (i.e., reduce the image size, lowering the resolution) and then encode (compress) the video to obtain low-resolution video data. This low-resolution video data is then transmitted over the network. Other electronic devices in the conference receive this low-resolution video data, decode (i.e., decompress) it, and then perform super-resolution processing to obtain high-resolution video data. This high-resolution video data has the same resolution as the captured video data, thus achieving high-definition video communication in the video conferencing scenario. This is understandable. Figure 1 The data transmission process is illustrated using a single frame from video data as an example. Figure 1 It can be seen that, in video conferencing scenarios, image super-resolution reconstruction equipment is an electronic device that receives video data.

[0033] For example, Figure 2 A flowchart illustrating an image super-resolution reconstruction method according to one embodiment of this application. (Reference) Figure 2 The image super-resolution reconstruction method specifically includes:

[0034] Step 110: Obtain a low-resolution image containing the target object.

[0035] Resolution can be understood as the number of pixels per inch of an image. A pixel is an indivisible unit in an image, meaning that an image is composed of pixels. Each pixel has a corresponding position (i.e., coordinate value) and an assigned color value (i.e., pixel value).

[0036] In one embodiment, the low-resolution image is the image that needs to be super-resolution reconstructed. The method of obtaining the low-resolution image is not limited. For example, the low-resolution image can be a single frame from video data. When performing super-resolution on the video data, each frame is processed as a unit, and in this case, each frame from the video data can be used as a low-resolution image.

[0037] In one embodiment, the low-resolution image contains a target object, which can be understood as the object to be super-resolution performed; that is, when super-resolution is performed on the low-resolution image, the target object is primarily super-resolution oriented. The type of target object is not limited and can be determined based on the usage scenario. For example, in a video call scenario, the target object is a human. It is understood that one or more types of target objects can be set according to actual needs. The number of target objects in the low-resolution image can be one or more. Optionally, when acquiring a low-resolution image, it is first determined whether it contains a target object. If it does, subsequent steps are performed; otherwise, no subsequent steps are performed. Then, a new frame of the low-resolution image can be acquired and this step can be repeated. For example, when the target object is a human, after acquiring the low-resolution image, it is first detected whether the low-resolution image contains a human image. If it does, subsequent steps are performed. The technical means for detecting whether a human image is contained are not limited in the embodiment; for example, neural network technology can be used to construct a neural network model for recognizing human images.

[0038] In one embodiment, after super-resolution is applied to the low-resolution image, a reconstructed image is obtained. For ease of distinction, this image is referred to as the super-resolution image. The size and resolution of the super-resolution image can be set according to the actual situation. Generally speaking, the resolution of the super-resolution image is higher than that of the low-resolution image.

[0039] Step 120: Extract the mask image of the target object from the low-resolution image.

[0040] A mask image of a target object can reveal the target object's position, shape, and other information in a low-resolution image. Optionally, the mask image can have two different pixel values: one for pixels representing the target object and another for pixels representing non-target objects. This allows for the differentiation of target and non-target objects using two different pixel values, thus clarifying the position and shape of the target object in the mask image.

[0041] In one embodiment, the method for extracting the mask image is not limited. For example, a segmentation network can be constructed using neural network technology. Then, the segmentation network processes the low-resolution image and outputs a mask image of the target object. The segmentation network can perform semantic prediction, that is, predict the pixels representing the target object in the low-resolution image, and then output the mask image of the target object. The type and structure of the segmentation network can be set according to the actual situation. It is understood that the segmentation network currently used belongs to the application stage of segmentation networks, and at this time, the segmentation network can output relatively stable results. Optionally, when using the segmentation network to process low-resolution images, the resolution of the mask image may be lower than the resolution of the low-resolution image. To facilitate subsequent processing, when the resolution of the mask image is lower than the resolution of the low-resolution image, interpolation is performed on the mask image. Interpolation can be understood as filling the gaps between pixels in the image to improve the image resolution. After interpolation, the number of pixels in the mask image can be increased, so that the mask image and the low-resolution image have the same resolution. The algorithm used for interpolation can be selected according to the actual situation, and this embodiment does not limit it.

[0042] Step 130: Based on the mask image, obtain the low-resolution region image of the target object in the low-resolution image.

[0043] For example, a mask image shows the position and shape of a target object in a low-resolution image. Therefore, the pixels representing the target object can be extracted from the low-resolution image using the mask image and combined to form a region image. The region image can be understood as the image of the area where the target object is located in the low-resolution image. In one embodiment, the region image representing the target object obtained from the low-resolution image is denoted as the low-resolution region image. The resolution of the low-resolution region image is the same as the resolution of the low-resolution image. Optionally, the low-resolution region image mainly displays the target object, omitting non-target objects from the low-resolution image or displaying a small number of non-target objects due to errors or other factors.

[0044] Optionally, a low-resolution region image can be obtained by performing a dot product between the mask image and the low-resolution image. It can be understood that each pixel in the mask image has only two pixel values: one representing the target object and the other representing a non-target object. During the dot product calculation, if a pixel in the low-resolution image is multiplied by a pixel in the mask image representing the target object, then that pixel in the low-resolution image can be considered as representing the target object and is retained after the dot product, forming the low-resolution region image. If a pixel in the low-resolution image is multiplied by a pixel in the mask image representing a non-target object, then that pixel in the low-resolution image can be considered as representing the non-target object and is retained as a blank pixel after the dot product, or it may not be retained, or it may be assigned a fixed pixel value (such as 0). Alternatively, based on the pixels representing the target object in the mask image, the smallest bounding rectangle region containing the target object in the mask image is determined. Then, the mask image within the smallest bounding rectangle region is dot-producted with the low-resolution image to obtain the low-resolution region image. The dot-product calculation method is the same as that described above for the dot-product calculation of the mask image and the low-resolution image, and will not be repeated here. It can be understood that the low-resolution region image obtained by using the dot-product of the mask image and the low-resolution image within the smallest bounding rectangle region is smaller in size than the low-resolution region image obtained by dot-product of the mask image and the low-resolution image.

[0045] Step 140: Use a super-resolution neural network to process the low-resolution region image to obtain the super-resolution region image of the target object.

[0046] Super-resolution neural networks are pre-built neural networks that can be used for super-resolution, and their type and structure can be set according to actual conditions. In one embodiment, the super-resolution neural network is denoted as SRNet, and the backbone network of SRNet adopts the fsrcnn structure. Here, fsrcnn is a convolutional neural network that can be used for super-resolution reconstruction.

[0047] Understandably, the current stage of using SRNet to obtain super-resolution region images belongs to the application stage of SRNet, where SRNet can output relatively stable results. Before applying SRNet, it needs to be trained and tested. Training is used to enable SRNet to have predictive capabilities (i.e., super-resolution reconstruction capabilities) and output stable results, while testing is used to test the accuracy of SRNet's output. The dataset and loss function used for training and testing SRNet can be selected according to the actual situation. For example, the public dataset DIV2K can be used to construct the training dataset for SRNet training and the test dataset for testing, and the L1 loss function can be used as the loss function for SRNet. The L1 loss function is also called the L1 norm loss function, which minimizes the sum of the absolute differences between the target value and the estimated value. Understandably, the device used to train and test SRNet can be an image super-resolution reconstruction device or other electronic devices. When the device used for training and testing is an image super-resolution reconstruction device, after training and testing are completed, SRNet can be directly deployed on the image super-resolution reconstruction device for application. When the training and testing devices are other electronic devices, after training and testing are completed, the SRNet in the other electronic devices needs to be deployed in the image super-resolution reconstruction device for application.

[0048] In one embodiment, the super-resolution neural network can output an image at a set resolution, which is the resolution of the super-resolution image. In another embodiment, after a low-resolution region image is input into the super-resolution neural network, the network performs super-resolution on the low-resolution region image and outputs a high-resolution image. The image output by the super-resolution neural network is denoted as the super-resolution region image. Since the low-resolution region image primarily displays the target object, the super-resolution region image can be understood as the result of super-resolution on the region containing the target object. It is understood that the resolution of the super-resolution region image is higher than the resolution of the low-resolution region image.

[0049] Step 150: Perform an upsampling operation on the low-resolution image to obtain an upsampled image.

[0050] It is understandable that the super-resolution region image obtained using a super-resolution neural network is the result of super-resolutioning the region where the target object is located in the low-resolution image. The final super-resolution image should include not only the region where the target object is located but also the regions in the low-resolution image where the target object is not located. Therefore, it is also necessary to obtain the super-resolution image of the regions in the low-resolution image where the target object is not located. In one embodiment, the low-resolution image is enlarged and its resolution increased by upsampling. During upsampling, the resolution of the super-resolution image is used as the target, and the low-resolution image is upsampled to make the resolution of the upsampled low-resolution image the same as the resolution of the super-resolution image. The upsampled low-resolution image can be referred to as the upsampled image. In one embodiment, the low-resolution image is upsampled by interpolation, that is, new elements (i.e., pixels) are inserted between pixels in the low-resolution image using a suitable interpolation algorithm. The interpolation algorithm can be set according to the actual situation. By using upsampled images to provide a reference for regions in a super-resolution image where the non-target object is located, it can be understood that the accuracy of the upsampled image is lower than that of the super-resolution region image obtained based on a super-resolution neural network.

[0051] It is understood that steps 150 and steps 120-140 can be executed simultaneously or sequentially, and the embodiments do not limit this.

[0052] Step 160: Fuse the upsampled image and the super-resolution region image to obtain the super-resolution image of the low-resolution image. The upsampled image, the super-resolution region image, and the super-resolution image have the same resolution.

[0053] In one embodiment, the upsampled image, the super-resolution region image, and the super-resolution image have the same resolution. By fusing the upsampled image and the super-resolution region image, the super-resolution image can be obtained.

[0054] For example, image fusion uses a specific algorithm to combine two or more images into a new image. In one embodiment, during image fusion, the region containing the target object in the super-resolution region image and the region not containing the target object in the upsampled image can be referenced to obtain the final super-resolution image. The algorithm used to fuse the upsampled image and the super-resolution region image can be selected according to the actual situation. For example, during image fusion, for a pixel position in the super-resolution image, if it is determined that the pixel at the same position in the super-resolution region image is the region where the target object is located, then the pixel value at that position in the super-resolution region image is directly used as the pixel value at that position in the super-resolution image; if it is determined that the pixel at the same position in the super-resolution region image is not the region where the target object is located, then the pixel value at that position in the upsampled image is directly used as the pixel value at that position in the super-resolution image. As another example, a weight map can be set, which has the same resolution as the upsampled image, the super-resolution region image, and the super-resolution image. Each pixel in the weight map corresponds to a weight value, ranging from [0-1]. During image fusion, the weight value corresponding to a pixel in the weight map can be used by pixels at the same location in the upsampled image, and the difference between 1 and the weight value can be used by pixels at the same location in the super-resolution region image. In other words, the weight value determines the parameter ratio of pixel values ​​in the upsampled image and the reference ratio of pixel values ​​in the super-resolution region image during fusion, thus obtaining the pixel values ​​in the super-resolution image. In one embodiment, the weight map can be determined using a mask image. For example, after interpolating the mask image, it can be upsampled to the same resolution as the super-resolution image. Then, the upsampled image can be processed using Gaussian blur to obtain the weight map.

[0055] Optionally, if the low-resolution region image is obtained by performing a dot product calculation between the minimum bounding rectangle region containing the target object in the mask image and the low-resolution image, then the size of the corresponding super-resolution region image will also be smaller than the size of the super-resolution image. That is, the total number of pixels contained in the super-resolution region image is less than the total number of pixels in the super-resolution image. In this case, in order to ensure that the super-resolution region image used during fusion has the same total number of pixels as the super-resolution image (both horizontal and vertical pixels are the same), when obtaining the super-resolution region image, it is also necessary to perform edge expansion on the super-resolution region image, that is, to add blank pixels to the super-resolution region image.

[0056] It is understandable that when low-resolution images are images in video data, processing each frame of low-resolution images in the video data according to the above image super-resolution reconstruction method can yield video data with higher resolution, that is, clearer video data.

[0057] The above-described technique, which involves acquiring a low-resolution image containing the target object, obtaining a mask image containing the target object based on the low-resolution image, determining the low-resolution region image of the target object's location within the low-resolution image based on the mask image, using a super-resolution neural network to obtain the corresponding super-resolution region image, and then upsampling the low-resolution image to obtain an upsampled image. Finally, the super-resolution region image and the upsampled image are fused to obtain the final super-resolution image. This technique addresses the slow processing speed issue in video scenarios when performing super-resolution in related technologies. Compared to using a super-resolution neural network to perform super-resolution on all low-resolution images, using the target object's mask image can extract the region containing the target object from the low-resolution image. By performing super-resolution only on the target object using the super-resolution neural network, the data processing load of the super-resolution neural network can be reduced, improving its processing speed and computational efficiency. Furthermore, by upsampling the low-resolution image, the resolution of regions not containing the target object is improved, ensuring that the final super-resolution image not only contains a clearer target object but also includes non-target objects.

[0058] Figure 3 A flowchart illustrating another image super-resolution reconstruction method provided in one embodiment of this application. Figure 3 The image super-resolution reconstruction method shown is a specific implementation based on the aforementioned image super-resolution reconstruction method. (Reference) Figure 3 The image super-resolution reconstruction method includes:

[0059] Step 210: Obtain a low-resolution image containing the target object.

[0060] Step 220: Use a segmentation network to process the low-resolution image to obtain the mask image of the target object.

[0061] A segmentation network is a pre-built neural network used to extract mask images. It can segment low-resolution images based on their semantics to output mask images of target objects within those images. The type and structure of the segmentation network can be determined according to specific circumstances. In one embodiment, a U-net network is used as the segmentation network. The U-net network can be divided into two parts: the first part extracts image features, and the second part restores those features; essentially, the U-net network can be understood as an encoder-decoder.

[0062] It's understandable that the current stage of using a segmentation network to obtain a mask image belongs to the application stage of the segmentation network. At this stage, the segmentation network can output relatively stable results. Before applying the segmentation network, it needs to be trained and tested. Training is used to enable the segmentation network to have segmentation capabilities and output stable segmentation results, while testing is used to test the accuracy of the segmentation network's output. The dataset and loss function used for training and testing the segmentation network can be selected according to the actual situation. For example, when the target object is a human, the publicly available human image segmentation dataset Supervisely can be used to construct the training dataset for training and the test dataset for testing, and the cross-entropy loss function can be used as the loss function for the segmentation network. It can be understood that the device used for training and testing the segmentation network can be an image super-resolution reconstruction device or other electronic devices. When the device used for training and testing is an image super-resolution reconstruction device, the segmentation network can be directly deployed on the image super-resolution reconstruction device after training and testing are completed. When the device used for training and testing is other electronic devices, the segmentation network on other electronic devices needs to be deployed on the image super-resolution reconstruction device after training and testing are completed. Optionally, the segmentation network and SRNet can be trained and tested on the same device or on different devices.

[0063] In one embodiment, a low-resolution image is input into a segmentation network for processing to obtain a mask image of the target object.

[0064] Step 230: When the resolution of the mask image is less than that of the low-resolution image, perform an upsampling operation on the mask image. The upsampled mask image and the low-resolution image will have the same resolution.

[0065] Generally, when a segmentation network processes an image, the resolution of the output image decreases; that is, the resolution of the output mask image is lower than that of the low-resolution image. In this case, the total number of pixels in the mask image is lower than the total number of pixels in the low-resolution image. Therefore, it is necessary to upsample the mask image to make its resolution the same as that of the low-resolution image, meaning both have the same total number of pixels (both horizontal and vertical). In one embodiment, a bicubic interpolation algorithm is used to implement the upsampling process. Bicubic interpolation, also called bicubic interpolation, is a method used to "interpolate" or increase the number / density of "pixels" in an image.

[0066] Optionally, after the segmentation network outputs the mask image, bicubic interpolation can be directly performed on the mask image to make the resolution of the mask image the same as the resolution of the low-resolution image. Alternatively, after the segmentation network outputs the mask image, it first determines whether the resolution of the mask image is less than the resolution of the low-resolution image. If it is less, bicubic interpolation is performed on the mask image to make the resolution of the mask image the same as the resolution of the low-resolution image; otherwise, this step is skipped, and subsequent steps are executed directly.

[0067] Step 240: Based on the mask image, obtain the low-resolution region image of the target object in the low-resolution image.

[0068] In one embodiment, a low-resolution region image can be obtained through any of the following methods.

[0069] Option 1: Perform dot product calculation on the low-resolution image and the mask image to obtain a low-resolution region image of the target object.

[0070] For example, a dot product is performed between the low-resolution image and the mask image. This involves multiplying pixels at the same location in both images to ensure that the resulting low-resolution region image displays the area containing the target object from the low-resolution image, but displays little or no area containing non-target objects. For instance, if the mask image has pixels representing the target object with a value of 1 and pixels representing non-target objects with a value of 0, then by multiplying the mask image and the low-resolution image, the pixels representing the target object in the low-resolution image are retained, while the pixels representing non-target objects are set to 0, resulting in the low-resolution region image. In this case, the resolution of the low-resolution region image is the same as that of the low-resolution image, and the total number of pixels in the low-resolution region image is equal to the total number of pixels in the low-resolution image.

[0071] Option 2: Calculate the bounding rectangle of the region containing the target object in the mask image to obtain the bounding rectangle mask image; perform dot product calculation on the low-resolution image and the bounding rectangle mask image to obtain the low-resolution region image of the target object.

[0072] In one embodiment, calculating the bounding rectangle refers to finding the smallest bounding rectangle containing the region where the target object is located in the mask image. Optionally, the position of a pixel in the horizontal direction is denoted as x, and the position of a pixel in the vertical direction is denoted as y. By finding the smallest x and largest x in the horizontal direction and the smallest y and largest y in the vertical direction representing the target object in the mask image, the smallest bounding rectangle containing the region where the target object is located can be drawn based on the smallest x, largest x, smallest y, and largest y. At this time, the four sides of the smallest bounding rectangle contain the smallest x, largest x, smallest y, and largest y, respectively. After obtaining the smallest bounding rectangle, the image within the smallest bounding rectangle is extracted from the mask image, and this image is denoted as the bounding rectangle mask image. It can be understood that after extracting the bounding rectangle mask image, the positions of each pixel do not change; that is, the position of a pixel in the bounding rectangle mask image is the same as the position of that pixel in the mask image.

[0073] After obtaining the circumscribed rectangular mask image, the low-resolution image is multiplied by the circumscribed rectangular mask image to obtain the low-resolution region image. It can be understood that the dot product calculation only uses pixels in the low-resolution image that correspond to those in the circumscribed rectangular mask image; that is, pixels at the same position are used. The dot product calculation process is similar to that described in Scheme 1 and will not be repeated here. It should be noted that compared to obtaining the low-resolution region image through the mask image, the low-resolution region image obtained through the circumscribed rectangular mask image retains the same clarity, has a smaller size, and contains less content unrelated to the target object. Therefore, when using a super-resolution neural network to process the low-resolution region image, the computational load of the super-resolution neural network is reduced, which can improve the processing speed of the super-resolution neural network.

[0074] Step 250: Use a super-resolution neural network to process the low-resolution region image to obtain the super-resolution region image of the target object.

[0075] In one embodiment, when obtaining a low-resolution region image of the target object based on the dot product of a low-resolution image and an circumscribed rectangular mask image, the total number of pixels in the low-resolution region image is smaller than that in the low-resolution image. When obtaining a super-resolution region image based on the low-resolution region image using a super-resolution neural network, the total number of pixels in the super-resolution region image is less than that in the super-resolution image. In this case, in order to ensure that the super-resolution region image used during fusion has the same total number of pixels as the super-resolution image (i.e., the number of horizontal and vertical pixels are equal), when processing the low-resolution region image using a super-resolution neural network to obtain the super-resolution region image of the target object, the method further includes: edge augmentation of the super-resolution region image, wherein the super-resolution region image after edge augmentation has the same resolution as the super-resolution image.

[0076] In one embodiment, edge augmentation can be understood as expanding the edges of the super-resolution region image, that is, adding pixels representing non-target objects to the edges of the super-resolution region image. The pixel values ​​corresponding to the added pixels can be set according to actual conditions. In one embodiment, blank pixels can be added to represent non-target objects. Optionally, when obtaining the super-resolution region image, the position of each pixel in the super-resolution region image can be clearly defined. In this case, combining the horizontal and vertical pixel counts of the super-resolution image with the positions of each pixel in the super-resolution region image, edge augmentation is performed on the super-resolution region image to ensure that the augmented super-resolution region image and the original super-resolution image have the same horizontal and vertical pixel counts.

[0077] Step 260: Upsample the low-resolution image using a bicubic interpolation algorithm to obtain an upsampled image.

[0078] In one embodiment, a bicubic interpolation algorithm is used to upsample the low-resolution image to obtain an upsampled image with the same resolution as the super-resolution image.

[0079] Step 270: Perform an upsampling operation on the mask image to obtain a super-resolution mask image. The super-resolution mask image and the super-resolution image have the same resolution.

[0080] For example, an upsampling operation is performed on the mask image to enlarge it. The resolution of the enlarged mask image is the same as that of the upsampled image and the super-resolution image, and the total number of pixels is equal. It is understood that the mask image has the same resolution as the low-resolution image; therefore, the mask image can be upsampled in the same way as the low-resolution image. In one embodiment, a bicubic interpolation algorithm is used to upsample the mask image, and the upsampled mask image is recorded as the super-resolution mask image.

[0081] Step 280: Use a Gaussian kernel to blur the super-resolution mask image to obtain a weight map, where each pixel in the weight map corresponds to a weight value.

[0082] The Gaussian kernel, also known as the Gaussian kernel function, is a radial basis function. In image processing, it can be used to blur images, effectively obscuring details. When using the Gaussian kernel for blurring, the value of the central pixel is the highest, while the weighting coefficients of surrounding pixels decrease as their distance from the central pixel increases. In one embodiment, after blurring the mask image using the Gaussian kernel, the values ​​of each pixel are between [0,1], and these values ​​can be used as weights during fusion. In another embodiment, the resulting blurred image is denoted as a weight map, where the value of each pixel is a weight.

[0083] Step 290: Use the weighted map to fuse the upsampled image and the super-resolution region image to obtain the super-resolution image of the low-resolution image.

[0084] In this system, the weighted map, upsampled image, super-resolution region image, and super-resolution image have the same resolution and the same total number of pixels. The pixel value of a pixel at a certain location in the super-resolution image can be determined by the weight value corresponding to that location in the weighted map, as well as the pixel value corresponding to that location in the upsampled image and the super-resolution region image.

[0085] In one embodiment, the fusion formula used for image fusion is:

[0086] HR_hat[i]=HR_BIC[i]*HR_Mask[i]+HR_HUMAN[i]*(1-HR_Mask[i])

[0087] Where HR_hat[i] represents the pixel value of the i-th pixel in the super-resolution image, HR_BIC[i] represents the pixel value of the i-th pixel in the upsampled image, HR_Mask[i] represents the weight value corresponding to the i-th pixel in the weight map, and HR_HUMAN[i] represents the pixel value of the i-th pixel in the super-resolution region image. HR_hat[i], HR_BIC[i], HR_Mask[i], and HR_HUMAN[i] have the same position. Following the above formula, the pixel values ​​of each pixel in the super-resolution image can be obtained, thus obtaining the super-resolution image.

[0088] The above image super-resolution reconstruction method is described below by way of example, wherein... Figure 4 This is a data flow diagram provided as an embodiment of this application. (Reference) Figure 4 In a video conferencing scenario where the target object is a human, after receiving video data transmitted from other electronic devices, the aforementioned image super-resolution reconstruction method is performed on each frame of the video data. Specifically, the current frame is treated as a low-resolution image containing the target object. Figure 4 The low-to-medium resolution images are denoted as LR. Then, the low-resolution image LR is input into the segmentation network. Figure 4 The segmentation network is denoted as Segnet. Then, the Segnet segmentation network is used to extract the mask image of the person (i.e., the target object). Figure 4 The mask image is denoted as Mask. The size of Mask is smaller than the size of the low-resolution image LR, and the resolution of Mask is smaller than the resolution of the low-resolution image LR. Then, the mask image is upsampled using bicubic interpolation to make the upsampled mask image have the same resolution as the low-resolution image. Figure 4The upsampled mask image is denoted as Human_Mask. Then, the bounding rectangle mask image of Human_Mask is calculated. Figure 4 The circumscribed rectangular mask image is denoted as Roi_Mask. Then, a dot product is calculated between the low-resolution image LR and the circumscribed rectangular mask image Roi_Mask to obtain the low-resolution region image of the human portrait. Figure 4 The low-to-medium resolution region image is denoted as Roi_LR. Then, the low-resolution region image Roi_LR is input into the super-resolution neural network. Figure 4 The super-resolution neural network is denoted as SRnet. SRnet is used to obtain the super-resolution region image corresponding to the low-resolution region image Roi_LR, and edge-filling is performed on the super-resolution region image. The filled super-resolution region image has the same resolution and the same total number of pixels as the original super-resolution image. Figure 4 The super-resolution region image after filling is denoted as HR_Human. Simultaneously, bicubic interpolation can be performed on the low-resolution image (LR) to obtain the upsampled image. Figure 4 The upsampled image is denoted as HR_BIC. The mask image Human_Mask is then upsampled to the same resolution as the super-resolution image. Finally, the upsampled mask image Human_Mask is fused with the upsampled image HR_BIC and the super-resolution region image HR_Human to obtain the super-resolution image. Figure 4 Medium-resolution images are denoted as

[0089] The above-described technique, which involves acquiring a low-resolution image containing the target object, using a segmentation network to obtain a mask image containing the target object, then determining the low-resolution region image of the target object's location within the low-resolution image based on the mask image, using a super-resolution neural network to obtain the corresponding super-resolution region image, and then upsampling the low-resolution image to obtain an upsampled image. Finally, a weight map is constructed based on the mask image, and the super-resolution region image and the upsampled image are fused based on the weight map to obtain the final super-resolution image. This technique solves the problem of slow processing speed in video scenarios when performing super-resolution. Compared to using a super-resolution neural network to perform super-resolution on all low-resolution images, using the target object's mask image can extract the region containing the target object from the low-resolution image, and then performing super-resolution only on the target object based on the super-resolution neural network. This reduces the data processing load of the super-resolution neural network, improves its processing speed and computational efficiency, and allows for the transmission of low-resolution video data in scenarios such as video conferencing, with super-resolution reconstruction performed at the receiving end, reducing transmission bandwidth and saving network resources. The segmentation network can accurately and quickly obtain the mask image. Image fusion can be achieved reasonably and quickly using weighted graphs to ensure the accuracy of super-resolution images. Furthermore, by calculating the bounding rectangle, only the region containing the target object can be processed, further reducing the computational load of the super-resolution image network and improving processing speed.

[0090] Figure 5 This is a schematic diagram of an image super-resolution reconstruction apparatus according to one embodiment of this application. (Reference) Figure 5 The image super-resolution reconstruction device includes: an image acquisition module 301, a mask extraction module 302, a region determination module 303, an image super-resolution module 304, an image upsampling module 305, and an image fusion module 306.

[0091] The system includes: an image acquisition module 301 for acquiring a low-resolution image containing the target object; a mask extraction module 302 for extracting the mask image of the target object from the low-resolution image; a region determination module 303 for obtaining a low-resolution region image of the target object from the low-resolution image based on the mask image; an image super-resolution module 304 for processing the low-resolution region image using a super-resolution neural network to obtain a super-resolution region image of the target object; an image upsampling module 305 for performing an upsampling operation on the low-resolution image to obtain an upsampled image; and an image fusion module 306 for fusing the upsampled image and the super-resolution region image to obtain a super-resolution image of the low-resolution image. The upsampled image, the super-resolution region image, and the super-resolution image have the same resolution.

[0092] In one embodiment of this application, the region determination module 303 is specifically used to: perform dot product calculation on the low-resolution image and the mask image to obtain a low-resolution region image of the target object.

[0093] In one embodiment of this application, the region determination module 303 includes: a rectangle calculation unit, used to calculate the bounding rectangle of the region where the target object is located in the mask image to obtain a bounding rectangle mask image; and a dot product unit, used to perform dot product calculation on the low-resolution image and the bounding rectangle mask image to obtain a low-resolution region image of the target object. Correspondingly, when the image super-resolution module 304 processes the low-resolution region image using a super-resolution neural network to obtain a super-resolution region image of the target object, it is also used to perform edge augmentation on the super-resolution region image, wherein the edge-augmented super-resolution region image has the same resolution as the super-resolution image.

[0094] In one embodiment of this application, the image fusion module 306 includes: a mask first upsampling unit, used to perform an upsampling operation on a mask image to obtain a super-resolution mask image, wherein the super-resolution mask image and the super-resolution image have the same resolution; a blurring unit, used to perform blurring processing on the super-resolution mask image using a Gaussian kernel to obtain a weight map, wherein each pixel in the weight map corresponds to a weight value; and a fusion unit, used to fuse the upsampled image and the super-resolution region image using the weight map to obtain a super-resolution image of the low-resolution image.

[0095] In one embodiment of this application, the mask extraction module 302 is specifically used to: process the low-resolution image using a segmentation network to obtain a mask image of the target object.

[0096] In one embodiment of this application, it further includes: a mask second upsampling module, used to process the low-resolution image using a segmentation network to obtain a mask image of the target object. When the resolution of the mask image is less than the resolution of the low-resolution image, the mask image is upsampled and the upsampled mask image has the same resolution as the low-resolution image.

[0097] In one embodiment of this application, the image upsampling module 305 is specifically used to: perform an upsampling operation on a low-resolution image using a bicubic interpolation algorithm to obtain an upsampled image.

[0098] The image super-resolution reconstruction apparatus provided above can be used to execute the image super-resolution reconstruction method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0099] It is worth noting that in the embodiments of the above-mentioned image super-resolution reconstruction device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy distinction between each other and are not used to limit the scope of protection of the present invention.

[0100] Figure 6 This is a schematic diagram of the structure of an image super-resolution reconstruction device provided in one embodiment of this application. Figure 6 As shown, the image super-resolution reconstruction device includes a processor 40, a memory 41, an input device 42, an output device 43, and a communication module 44; the number of processors 40 in the image super-resolution reconstruction device can be one or more. Figure 6 Taking a processor 40 as an example, the processor 40, memory 41, input device 42, output device 43, and communication module 44 in the image super-resolution reconstruction device can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0101] The memory 41, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the image super-resolution reconstruction method in this embodiment of the invention (e.g., the image acquisition module 301, mask extraction module 302, region determination module 303, image super-resolution module 304, image upsampling module 305, and image fusion module 306 in the image super-resolution reconstruction device). The processor 40 executes various functional applications and data processing of the image super-resolution reconstruction device by running the software programs, instructions, and modules stored in the memory 41, thereby realizing the above-described image super-resolution reconstruction method.

[0102] The memory 41 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created during the use of the image super-resolution reconstruction device. Furthermore, the memory 41 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 41 may further include memory remotely located relative to the processor 40, which can be connected to the image super-resolution reconstruction device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0103] Input device 42 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the image super-resolution reconstruction device. Output device 43 may include display devices such as a display screen. Communication device 44 can utilize a network for data communication, such as acquiring video data in a video conferencing scenario.

[0104] The aforementioned image super-resolution reconstruction equipment includes an image super-resolution reconstruction device, which can be used to perform any image super-resolution reconstruction method and has corresponding functions and beneficial effects.

[0105] Furthermore, embodiments of the present invention also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform relevant operations in the image super-resolution reconstruction method provided in any embodiment of the present application, and have corresponding functions and beneficial effects.

[0106] Those skilled in the art will understand that embodiments of this application may be provided as methods, systems, or computer program products.

[0107] Therefore, this application may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce implementations of the flowchart... Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0108] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0109] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0110] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0111] Note that the above description is merely a preferred embodiment of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of the present invention, the scope of which is determined by the scope of the appended claims.

Claims

1. An image super-resolution reconstruction method, characterized in that, include: Obtain a low-resolution image containing the target object; Extract the mask image of the target object from the low-resolution image; Based on the mask image, a low-resolution region image of the target object is obtained in the low-resolution image; The low-resolution region image is processed using a super-resolution neural network to obtain a super-resolution region image of the target object; The low-resolution image is upsampled to obtain an upsampled image; The upsampled image and the super-resolution region image are fused to obtain a super-resolution image of the low-resolution image, wherein the upsampled image, the super-resolution region image, and the super-resolution image have the same resolution; The process of fusing the upsampled image and the super-resolution region image to obtain the super-resolution image of the low-resolution image includes: An upsampling operation is performed on the mask image to obtain a super-resolution mask image, wherein the super-resolution mask image has the same resolution as the original super-resolution image; The super-resolution mask image is blurred using a Gaussian kernel to obtain a weight map, where each pixel in the weight map corresponds to a weight value. The upsampled image and the super-resolution region image are fused using the weighted map to obtain the super-resolution image of the low-resolution image.

2. The image super-resolution reconstruction method according to claim 1, characterized in that, The step of obtaining a low-resolution region image of the target object in the low-resolution image based on the mask image includes: The low-resolution image and the mask image are used to perform a dot product calculation to obtain a low-resolution region image of the target object.

3. The image super-resolution reconstruction method according to claim 1, characterized in that, The step of obtaining a low-resolution region image of the target object in the low-resolution image based on the mask image includes: The bounding rectangle of the region where the target object is located in the mask image is calculated to obtain the bounding rectangle mask image; The low-resolution image and the circumscribed rectangular mask image are used to perform a dot product calculation to obtain a low-resolution region image of the target object.

4. The image super-resolution reconstruction method according to claim 3, characterized in that, When processing the low-resolution region image using a super-resolution neural network to obtain the super-resolution region image of the target object, the method further includes: The super-resolution region image is edge-expanded, and the edge-expanded super-resolution region image has the same resolution as the super-resolution image.

5. The image super-resolution reconstruction method according to claim 1, characterized in that, The step of extracting the mask image of the target object from the low-resolution image includes: The low-resolution image is processed using a segmentation network to obtain a mask image of the target object.

6. The image super-resolution reconstruction method according to claim 5, characterized in that, After processing the low-resolution image using a segmentation network to obtain the mask image of the target object, the process further includes: When the resolution of the mask image is less than that of the low-resolution image, an upsampling operation is performed on the mask image, and the upsampled mask image has the same resolution as the low-resolution image.

7. The image super-resolution reconstruction method according to claim 1, characterized in that, The upsampling operation on the low-resolution image to obtain the upsampled image includes: The low-resolution image is upsampled using a bicubic interpolation algorithm to obtain an upsampled image.

8. An image super-resolution reconstruction device, characterized in that, include: The image acquisition module is used to acquire low-resolution images containing the target object. A mask extraction module is used to extract the mask image of the target object in the low-resolution image; A region determination module is used to obtain a low-resolution region image of the target object in the low-resolution image based on the mask image; An image super-resolution module is used to process the low-resolution region image using a super-resolution neural network to obtain a super-resolution region image of the target object. An image upsampling module is used to perform an upsampling operation on the low-resolution image to obtain an upsampled image; An image fusion module is used to fuse the upsampled image and the super-resolution region image to obtain a super-resolution image of the low-resolution image, wherein the upsampled image, the super-resolution region image, and the super-resolution image have the same resolution; The image fusion module includes: a mask first upsampling unit, used to perform upsampling operation on the mask image to obtain a super-resolution mask image, wherein the super-resolution mask image and the super-resolution image have the same resolution; The blurring unit is used to blur the super-resolution mask image using a Gaussian kernel to obtain a weight map, where each pixel in the weight map corresponds to a weight value; the fusion unit is used to fuse the upsampled image and the super-resolution region image using the weight map to obtain a super-resolution image of the low-resolution image.

9. An image super-resolution reconstruction device, characterized in that, include: One or more processors Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image super-resolution reconstruction method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the image super-resolution reconstruction method as described in any one of claims 1-7.

Images