Image processing method and device and storage medium

Abstract

The invention relates to an image processing method and device and a storage medium. The method comprises the following steps: acquiring a low-resolution feature map and a high-resolution feature map which are respectively extracted from the same image; performing linear projection on the high-resolution feature map to obtain a high-resolution projection feature map, and performing linear projection on the low-resolution feature map to obtain a low-resolution projection feature map; performing up-sampling on the low-resolution projection feature map to obtain a target key feature map, and determining a target query feature map based on the high-resolution projection feature map; for pixels in the target query feature map, determining neighborhood pixels corresponding to the pixels in the target key feature map, and performing differential convolution on the neighborhood pixels to obtain a convolution processing result; based on a convolution processing result, determining an up-sampling kernel corresponding to each pixel in the target query feature map; and performing up-sampling on the low-resolution feature map based on the up-sampling kernel to obtain an up-sampled feature map. By adopting the method, the accuracy of an up-sampling result can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device, and storage medium. Background Art

[0002] With the development of computer technology, feature upsampling technology has emerged. As a basic element in deep network architecture, feature upsampling aims to restore the spatial resolution of low-resolution (LR) features to obtain high-resolution (HR) features. Feature upsampling is widely used in network structures such as image segmentation and target detection tasks.

[0003] In related technologies, feature upsampling is performed by passing the LR features to be upsampled through an upsampling kernel prediction module to dynamically generate a pixel-by-pixel upsampling kernel. The upsampling kernel is then used to pass the original LR features through a content-aware reassembly module to obtain the upsampling result.

[0004] However, only generating the upsampling kernel from the LR features will result in it being unable to obtain relatively fidelity image detail information, resulting in low accuracy of the upsampling results. Summary of the Invention

[0005] Based on this, it is necessary to provide an image processing method, device and storage medium that can improve the accuracy of upsampling results in response to the above technical problems.

[0006] In a first aspect, the present application provides an image processing method. The method comprises:

[0007] Obtaining a low-resolution feature map and a high-resolution feature map respectively extracted from the same image, wherein the resolution of the high-resolution feature map is higher than that of the low-resolution feature map;

[0008] Performing linear projection on the high-resolution feature map to obtain a high-resolution projection feature map, and performing linear projection on the low-resolution feature map to obtain a low-resolution projection feature map;

[0009] Upsampling the low-resolution projected feature map according to the resolution of the high-resolution projected feature map to obtain a target key feature map, and determining a target query feature map based on the high-resolution projected feature map;

[0010] For a pixel in the target query feature map, determine a mirrored pixel at the same position as the pixel in the target key feature map, determine neighboring pixels of the mirrored pixel, perform differential convolution on the neighboring pixels, and obtain a convolution processing result corresponding to the pixel;

[0011] Determine an upsampling kernel corresponding to each pixel in the target query feature map based on a convolution processing result corresponding to each pixel in the target query feature map;

[0012] Based on the upsampling kernel corresponding to each pixel in the target query feature map, the low-resolution feature map is upsampled to obtain an upsampled feature map.

[0013] In a second aspect, the present application further provides an image processing device. The device comprises:

[0014] A feature map acquisition module, configured to acquire a low-resolution feature map and a high-resolution feature map respectively extracted from the same image, wherein the resolution of the high-resolution feature map is higher than that of the low-resolution feature map;

[0015] A feature map projection module is used to perform linear projection on the high-resolution feature map to obtain a high-resolution projection feature map, and to perform linear projection on the low-resolution feature map to obtain a low-resolution projection feature map;

[0016] a query feature map determining module, configured to upsample the low-resolution projected feature map according to the resolution of the high-resolution projected feature map to obtain a target key feature map, and determine a target query feature map based on the high-resolution projected feature map;

[0017] A differential convolution module is configured to determine, for a pixel in the target query feature map, a mirrored pixel at the same position as the pixel in the target key feature map, determine neighboring pixels of the mirrored pixel, perform differential convolution on the neighboring pixels, and obtain a convolution processing result corresponding to the pixel;

[0018] An upsampling kernel determining module, configured to determine an upsampling kernel corresponding to each pixel in the target query feature map based on a convolution processing result corresponding to each pixel in the target query feature map;

[0019] The feature map upsampling module is used to upsample the low-resolution feature map based on the upsampling kernel corresponding to each pixel in the target query feature map to obtain an upsampled feature map.

[0020] In some embodiments, the query feature map determination module is further used to project the high-resolution projection feature map into the semantic space where the low-resolution projection feature map is located to obtain a high-resolution alignment feature map that is aligned with the low-resolution projection feature map in the semantic space; and determine the high-resolution alignment feature map as the target query feature map.

[0021] In some embodiments, the query feature map determination module is also used to use the target key feature map as the input map and the high-resolution projection feature map as the guide map; perform guided filtering based on the guide map and the input map to obtain a high-resolution aligned feature map that is aligned with the low-resolution projection feature map in the semantic space.

[0022] In some embodiments, the differential convolution module is also used to obtain a preset hole coefficient, which is consistent with the value of a preset upsampling ratio; based on the hole coefficient, the neighborhood pixels of the mirror pixel are determined in the target key feature map; each neighborhood pixel in the target key feature map is subtracted from the pixel to obtain an updated neighborhood pixel; and hole convolution processing is performed on the updated neighborhood pixel to obtain a convolution processing result corresponding to the pixel.

[0023] In some embodiments, the image processing method of the present application also includes: a feature map denoising module, which is used to filter out noise on the high-resolution projection feature map to obtain a high-resolution denoised feature map; for a pixel in the high-resolution projection feature map, determine the neighboring pixels of its mirrored pixel in the high-resolution denoised feature map, perform differential convolution on the neighboring pixels in the high-resolution denoised feature map, and obtain the convolution processing results corresponding to the pixels in the high-resolution projection feature map; the upsampling kernel determination module is also used to: determine the upsampling kernel corresponding to each pixel in the target query feature map based on the convolution processing results corresponding to each pixel in the target query feature map and the convolution processing results corresponding to each pixel in the high-resolution projection feature map.

[0024] In some embodiments, the upsampling kernel determination module is further used to: determine the similarity representation value corresponding to each pixel in the target query feature map based on the convolution processing result corresponding to each pixel in the target query feature map; determine the similarity representation value corresponding to each pixel in the high-resolution projection feature map based on the convolution processing result corresponding to each pixel in the high-resolution projection feature map; perform statistics on the similarity representation values ​​of pixels at the same position in the target query feature map and the high-resolution projection feature map, and determine the upsampling kernel corresponding to each pixel in the target query feature map based on the statistical results.

[0025] In some embodiments, the feature map denoising module is further used to determine a Gaussian blur kernel; based on the Gaussian blur kernel, Gaussian blur processing is performed on the high-resolution projection feature map to obtain a high-resolution denoised feature map.

[0026] In some embodiments, the feature map denoising module is further used to: determine, for a pixel in the high-resolution projection feature map, its mirror pixel in the high-resolution denoising feature map; determine, based on a preset hole coefficient, the neighborhood pixels of the mirror pixel in the high-resolution denoising feature map, wherein the hole coefficient is consistent with the value of a preset upsampling ratio; subtract each neighborhood pixel in the high-resolution denoising feature map from the pixel to obtain an updated neighborhood pixel; perform hole convolution processing on the updated neighborhood pixel to obtain a convolution processing result corresponding to the pixel in the high-resolution projection feature map.

[0027] In some embodiments, the feature map upsampling module is further used to: perform bilinear interpolation on the low-resolution feature map according to the resolution of the high-resolution projection feature map to obtain an intermediate feature map; and upsample the intermediate feature map based on the upsampling kernel corresponding to each pixel in the target query feature map to obtain an upsampled feature map.

[0028] In some embodiments, the feature map upsampling module is further used to: for a pixel in the target query feature map, determine the target neighborhood pixel corresponding to the pixel in the intermediate feature map, and the target neighborhood pixel has the same position as the neighborhood pixel of the pixel in the target key feature map; based on the upsampling kernel corresponding to the pixel, perform weighted summation on the target neighborhood pixel to obtain the upsampling pixel corresponding to the pixel; the upsampling pixels corresponding to each pixel in the target query feature map are used to form an upsampling feature map.

[0029] In some embodiments, the feature map acquisition module is further used to acquire the image to be segmented, extract shallow features from the image to be segmented, and obtain a high-resolution feature map; and input the image to be segmented into the image segmentation network to obtain a low-resolution feature map.

[0030] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned image processing method when executing the computer program.

[0031] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned image processing method when executed by a processor.

[0032] In a fifth aspect, the present application further provides a computer program product, which includes a computer program that implements the steps of the above-mentioned image processing method when executed by a processor.

[0033] The above-mentioned image processing method, apparatus, computer device, storage medium and computer program product obtain a low-resolution feature map and a high-resolution feature map extracted from the same image, linearly project the high-resolution feature map to obtain a high-resolution projection feature map, linearly project the low-resolution feature map to obtain a low-resolution projection feature map, upsample the low-resolution projection feature map according to the resolution of the high-resolution projection feature map to obtain a target key feature map, determine a target query feature map based on the high-resolution projection feature map, for a pixel in the target query feature map, determine a mirrored pixel with the same position as the pixel in the target key feature map, determine the neighboring pixels of the mirrored pixel, and perform differential analysis on the neighboring pixels. The convolution operation is performed to obtain the convolution processing result corresponding to the pixel. Based on the convolution processing result corresponding to each pixel in the target query feature map, the upsampling kernel corresponding to each pixel in the target query feature map is determined. Based on the upsampling kernel corresponding to each pixel in the target query feature map, the low-resolution feature map is upsampled to obtain the upsampled feature map. This can guide the upsampling process of the low-resolution feature map by using the high-resolution feature map, help restore the details in the upsampling process of the low-resolution feature map, and make the feature map obtained by upsampling more accurate. The upsampling kernel can be determined by differential convolution, which can flexibly capture the true local similarity, further improving the accuracy of the feature map obtained by upsampling. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A diagram showing an application environment of an image processing method in some embodiments;

[0036] Figure 2 A schematic diagram of the overall process of the image processing method in some embodiments;

[0037] Figure 3 is a flowchart of an image processing method in some embodiments;

[0038] Figure 4 A schematic diagram of a flow chart of the application of the image processing method of the present application to an image segmentation task in some embodiments;

[0039] Figure 5 is a flowchart of an image processing method in some other embodiments;

[0040] Figure 6 This is a diagram of the overall framework for upsampling using the image processing method of the present application in some embodiments;

[0041] Figure 7 Schematic diagram of a framework of a paired central difference convolution module in some embodiments;

[0042] Figure 8 A schematic diagram of the implementation logic of the image segmentation task in some embodiments;

[0043] Figure 9 A schematic diagram showing a comparison of visual effects of image semantic segmentation in some embodiments;

[0044] Figure 10 A schematic diagram showing a comparison of performance indicators in applying the image processing method of the present application to an image segmentation task in some embodiments;

[0045] Figure 11 A schematic diagram of a process of applying the image processing method of the present application to an image detection task in some embodiments;

[0046] Figure 12 is a structural block diagram of an image processing device in some embodiments;

[0047] Figure 13 is a diagram of the internal structure of a computer device in some embodiments;

[0048] Figure 14 1 is a diagram of the internal structure of a computer device in some other embodiments. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0050] The image processing method provided in the embodiments of the present application involves technologies such as machine learning (ML) and computer vision (CV) in artificial intelligence, wherein:

[0051] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0052] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0053] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, which uses cameras and computers to replace the human eye in identifying and measuring objects, and then further processes the images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Large model technology has brought significant changes to the development of computer vision technology. Pre-trained models in the field of vision, such as the Swin Transformer, ViT, V-MOE, and MAE, can be fine-tuned to quickly and widely apply to specific downstream tasks. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and other technologies. It also includes common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0054] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning. Pretrained models are the latest development in deep learning, integrating these techniques.

[0055] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence generated content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0056] The image processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 and the server 104 can communicate through a network, such as a wired or wireless network. The data storage system can store data that the server 104 needs to process, for example, it can store image data. The data storage system can be set up separately, can be integrated on the server 104, or can be placed on the cloud or other servers. The terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0057] The image processing method provided in the embodiment of the present application can be performed by a computer device, which refers to an electronic device with data calculation, processing and storage capabilities. Figure 1 Taking the application environment shown as an example, the image processing method can be executed by the terminal 102 alone, or by the server 104 alone, or by the terminal 102 and the server 104 interacting and cooperating to execute, which is not limited in this application.

[0058] Take the server 104 as an example, Figure 2 The server can obtain a low-resolution feature map and a high-resolution feature map extracted from the same image, perform linear projection on the high-resolution feature map to obtain a high-resolution projection feature map, perform linear projection on the low-resolution feature map to obtain a low-resolution projection feature map, upsample the low-resolution projection feature map according to the resolution of the high-resolution projection feature map to obtain a target key feature map, determine the target query feature map based on the high-resolution projection feature map, for a pixel in the target query feature map, determine the mirror pixel with the same position as the pixel in the target key feature map, and determine the neighboring pixels of the mirror pixel, perform differential convolution on the neighboring pixels to obtain the convolution processing result corresponding to the pixel, determine the upsampling kernel corresponding to each pixel in the target query feature map based on the convolution processing result corresponding to each pixel in the target query feature map, upsample the low-resolution feature map based on the upsampling kernel corresponding to each pixel in the target query feature map to obtain an upsampled feature map.

[0059] In some embodiments, as Figure 3 As shown, an image processing method is provided, which is executed by a computer device, which may be Figure 1 In the server 104 or terminal 102, in the embodiment of the present application, the method is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0060] Step 302: Obtain a low-resolution feature map and a high-resolution feature map extracted from the same image. The resolution of the high-resolution feature map is higher than that of the low-resolution feature map.

[0061] Among them, the low-resolution feature map refers to a feature map with lower resolution, and the high-resolution feature map refers to a feature map with higher resolution. The resolution of the high-resolution feature map is higher than that of the low-resolution feature map. The low-resolution feature map and the high-resolution feature map are feature maps extracted from the same image. The low-resolution feature map is the feature map to be upsampled, and the high-resolution feature map is used to provide guiding feature information during the upsampling process of the feature map to be upsampled, so as to guide the upsampling process of the feature map to be upsampled.

[0062] Optionally, the server can perform shallow feature extraction on the image to obtain a high-resolution feature map, and then input the image into the main network to obtain a low-resolution feature map. The main network can be a network based on a deep neural network architecture. In different application scenarios, the main network can be different networks. For example, in an object detection application scenario, the main network can be an object detection network.

[0063] In step 304 , linear projection is performed on the high-resolution feature map to obtain a high-resolution projection feature map, and linear projection is performed on the low-resolution feature map to obtain a low-resolution projection feature map.

[0064] Specifically, the server can perform linear projection on the high-resolution feature map and the low-resolution feature map respectively. After linear projection on the high-resolution feature map, a high-resolution projection feature map can be obtained. After linear projection on the low-resolution feature map, a low-resolution projection feature map can be obtained. The number of channels of the obtained high-resolution projection feature map and the low-resolution feature map are both smaller than the image before projection, that is, the low-resolution feature map and the high-resolution feature map can be reduced in dimension through feature projection, and the flexibility of the features can be improved, so that the sampling results obtained by feature upsampling are more accurate.

[0065] For example, the server may perform linear projection on the high-resolution feature map and the low-resolution feature map respectively through group normalization and a 1*1 convolution layer.

[0066] Step 306 : Upsample the low-resolution projected feature map according to the resolution of the high-resolution projected feature map to obtain a target key feature map, and determine a target query feature map based on the high-resolution projected feature map.

[0067] Among them, upsampling refers to the process of increasing the resolution of the feature map.

[0068] Specifically, the server can upsample the low-resolution projection feature map according to the resolution of the high-resolution projection feature map, so that the target key feature map is consistent with the resolution of the low-resolution projection feature map. For example, the server can perform bilinear interpolation upsampling on the low-resolution projection feature map according to the resolution of the high-resolution projection feature map to obtain the target key feature map. The server can also determine the target query feature map based on the high-resolution projection feature map. The resolution of the target query feature map is the same as that of the high-resolution feature map. Figure 1 The upsampling kernel of each pixel in the high-resolution feature map can be determined based on the target query feature map and the target key feature map, and then the low-resolution feature map can be upsampled based on the upsampling kernel to obtain the final upsampling feature map.

[0069] Optionally, the server may directly determine the high-resolution projected feature map as the target query feature map. Also optionally, the server may align the high-resolution projected feature map with the low-resolution projected feature map in the semantic space and determine the aligned high-resolution projected feature map as the target query feature map.

[0070] Step 308: For a pixel in the target query feature map, determine a mirrored pixel with the same position as the pixel in the target key feature map, determine the neighborhood pixels of the mirrored pixel, perform differential convolution on the neighborhood pixels, and obtain a convolution processing result corresponding to the pixel.

[0071] Among them, the target query feature map and the target key feature map are feature maps with the same resolution, so that for pixel i in the target query feature map, its mirror pixel in the target key feature map is the pixel at the same position as the pixel i. For example, assuming that pixel i in the target query feature map is the pixel in the 1st row and 4th column of the target query feature map, then the mirror pixel of pixel i in the target key feature map is the pixel in the 1st row and 4th column of the target key feature map.

[0072] Specifically, for pixel i in the target query feature map, the server can determine the mirror pixel with the same position as pixel i in the target key feature map, and then determine at least one neighborhood pixel of the mirror pixel in the target key feature map, and perform differential convolution on these neighborhood pixels with pixel i as the center pixel, so as to obtain the convolution processing result corresponding to pixel i, wherein the differential convolution is to subtract the neighborhood pixel from pixel i to update the neighborhood pixel, and then perform convolution processing on the updated neighborhood pixel.

[0073] Optionally, the server can also process the target query feature map and the target key feature map through a group normalization layer (group Normalization) respectively, and then when performing differential convolution, the differential convolution can be performed based on the normalized target key feature map and the target query feature map, that is, for pixel i in the target query feature map, the mirror pixel of the pixel i is determined in the normalized target key feature map, and the neighborhood pixels of the mirror pixel are determined in the normalized target key feature map, and each neighborhood pixel is subtracted from the normalized pixel i to update the neighborhood pixel, and then the updated neighborhood pixel is convolved, so that the convolution processing result corresponding to pixel i can be obtained.

[0074] Optionally, when determining neighborhood pixels, the server may use a hole coefficient that is consistent with the value of a preset upsampling ratio to determine the neighborhood pixels, thereby ensuring that the selected neighborhood pixels can contain more semantics and further improving the accuracy of upsampling.

[0075] Step 310 : Determine an upsampling kernel corresponding to each pixel in the target query feature map based on the convolution processing result corresponding to each pixel in the target query feature map.

[0076] Usually, an upsampled high-resolution pixel is obtained by weighted summation of the low-resolution pixels around it, and this weighting coefficient is the upsampling kernel.

[0077] Specifically, the server can determine the similarity representation information corresponding to each pixel in the target query feature map based on the convolution processing results corresponding to each pixel in the target query feature map. The similarity representation information corresponding to each pixel contains K 2 channels, each channel represents the similarity between the pixel and a neighboring pixel, and then the upsampling kernel corresponding to each pixel in the target query feature map is determined based on the similarity representation information corresponding to each pixel in the target query feature map.

[0078] It can be understood that since the resolution of the target query feature map is the same as that of the high-resolution feature map extracted from the image, the upsampling kernel corresponding to pixel i in the target query feature map is the upsampling kernel corresponding to pixel i in the high-resolution feature map.

[0079] Step 312 : upsample the low-resolution feature map based on the upsampling kernel corresponding to each pixel in the target query feature map to obtain an upsampled feature map.

[0080] It can be understood that since the resolution of the target query feature map is the same as that of the high-resolution feature map extracted from the image, the upsampling kernel corresponding to pixel i in the target query feature map is the upsampling kernel corresponding to pixel i in the high-resolution feature map.

[0081] Specifically, the server can upsample the low-resolution feature map based on the upsampling kernel corresponding to each pixel in the target query feature map to obtain the upsampled pixels corresponding to each pixel, and then obtain the upsampled feature map.

[0082] Taking pixel i in the target query feature map as an example, the server can determine the target neighborhood pixels corresponding to pixel i in the high-resolution feature map based on the low-resolution feature map. The target neighborhood pixels correspond one-to-one to the upsampling kernel values ​​in the upsampling kernel of pixel i, and the relative position relationship between the target neighborhood pixels corresponding to each upsampling kernel value in the upsampling kernel and the pixel i is consistent with the relative position relationship between the neighborhood pixels and the pixel i used in the calculation process of the upsampling coefficient. The server further performs weighted summation on each value in the upsampling kernel of the pixel i and their corresponding target neighborhood pixels to obtain the upsampling pixel of the pixel i.

[0083] Optionally, since the low-resolution projection feature map is upsampled according to the resolution of the high-resolution projection feature map in the process of determining the upsampling kernel, the determined neighborhood pixels are located based on the size of the high-resolution feature map. Then, in the process of upsampling according to the upsampling kernel, the neighborhood pixels at the same position as in the process of determining the upsampling kernel can be determined, thereby further improving the accuracy of upsampling. Based on this, the server can preliminarily upsample the low-resolution feature map according to the resolution of the high-resolution feature map, so that the size of the low-resolution feature map is consistent with the size of the high-resolution feature map. Figure 1 The server can then determine the target neighborhood pixels of pixel i in the high-resolution feature map in the feature map obtained by preliminary upsampling, and then perform weighted summation of these target neighborhood pixels through the upsampling kernel to obtain the high-resolution feature map corresponding to pixel i.

[0084] In the above image processing method, a low-resolution feature map and a high-resolution feature map are extracted from the same image respectively, a high-resolution feature map is linearly projected to obtain a high-resolution projection feature map, a low-resolution feature map is linearly projected to obtain a low-resolution projection feature map, the low-resolution projection feature map is upsampled according to the resolution of the high-resolution projection feature map to obtain a target key feature map, a target query feature map is determined based on the high-resolution projection feature map, for a pixel in the target query feature map, a mirrored pixel with the same position as the pixel is determined in the target key feature map, and the neighboring pixels of the mirrored pixel are determined, and differential convolution is performed on the neighboring pixels to obtain the pixel corresponding to the target key feature map. The convolution processing result is based on the convolution processing result corresponding to each pixel in the target query feature map, and the upsampling kernel corresponding to each pixel in the target query feature map is determined. Based on the upsampling kernel corresponding to each pixel in the target query feature map, the low-resolution feature map is upsampled to obtain the upsampled feature map, so that the high-resolution feature map can be used to guide the upsampling process of the low-resolution feature map, which can help restore the details in the upsampling process of the low-resolution feature map, making the feature map obtained by upsampling more accurate, and the upsampling kernel can be determined by differential convolution, which can flexibly capture the true local similarity, further improving the accuracy of the feature map obtained by upsampling.

[0085] In some embodiments, determining a target query feature map based on a high-resolution projected feature map includes: projecting the high-resolution projected feature map into the semantic space where the low-resolution projected feature map is located to obtain a high-resolution aligned feature map that is aligned with the low-resolution projected feature map in the semantic space; and determining the high-resolution aligned feature map as the target query feature map.

[0086] Specifically, the server can use the semantic information of the low-resolution projection feature map as a reference to align the semantics of the high-resolution projection feature map with the semantic information of the low-resolution projection feature map. In the specific alignment process, the server can project the high-resolution projection feature map into the semantic space where the low-resolution projection feature map is located. The feature map obtained after the projection is the feature map that is aligned with the semantic information of the low-resolution projection feature map. The server can further use the high-resolution aligned feature map as the target query feature map.

[0087] For example, projecting the high-resolution projection feature map to the semantic space where the low-resolution projection feature map is located can be achieved through a machine learning model, and the training samples of the machine learning model can be two exactly the same images, or two images semantically aligned through guided filtering. The training goal of the machine learning model is to make the similarity between the two processed images as small as possible.

[0088] In this embodiment, the high-resolution projected feature map is projected into the semantic space where the low-resolution projected feature map is located, thereby obtaining a target query feature map. Due to the semantic alignment, the semantic information represented by the subsequently determined neighborhood features is closer to the semantic information of the pixels in the high-resolution feature map serving as the guide feature map, thereby further improving the accuracy of upsampling.

[0089] In some embodiments, the high-resolution projection feature map is projected into the semantic space where the low-resolution projection feature map is located to obtain a high-resolution aligned feature map that is aligned with the low-resolution projection feature map in the semantic space, including: using the target key feature map as the input map and the high-resolution projection feature map as the guide map; performing guided filtering based on the guide map and the input map to obtain a high-resolution aligned feature map that is aligned with the low-resolution projection feature map in the semantic space.

[0090] Among them, guided filtering is an image filtering technology that filters the initial image p (input image) through a guiding image I, so that the final output image is roughly similar to the initial image P, but the texture part is similar to the guiding image I.

[0091] Specifically, the server can use the code toolkit for guided filtering, use the high-resolution projection feature map and the target key feature map as the guide map and input map in the guided filtering respectively, and output the filtering result. The filtering output result is a high-resolution aligned feature map that is aligned with the low-resolution projection feature map in the semantic space.

[0092] In the above embodiment, by taking the target key feature map as the input map and the high-resolution projection feature map as the guide map, and performing guided filtering based on the guide map and the input map, the high-resolution projection feature map can be quickly projected into the semantic space where the low-resolution projection feature map is located, thereby improving the image processing efficiency.

[0093] In some embodiments, the neighborhood pixels of the mirrored pixel are determined, differential convolution is performed on the neighborhood pixels, and a convolution processing result corresponding to the pixel is obtained, including: obtaining a preset hole coefficient, the hole coefficient is consistent with the numerical value of the preset upsampling ratio; based on the hole coefficient, the neighborhood pixels of the mirrored pixel are determined in the target key feature map; each neighborhood pixel in the target key feature map is subtracted from the pixel to obtain an updated neighborhood pixel; and hole convolution is performed on the updated neighborhood pixel to obtain a convolution processing result corresponding to the pixel.

[0094] The preset upsampling ratio refers to the preset upsampling ratio for low-resolution features. The hole coefficient is consistent with the value of the preset upsampling ratio, that is, the hole coefficient is a value greater than 1.

[0095] The low-resolution projection feature map is upsampled according to the resolution of the high-resolution projection feature map to obtain the target key feature map to participate in the calculation of the upsampling kernel. After upsampling, the semantics of a pixel in the low-resolution feature map is expressed by multiple pixels in the target key feature map. By increasing the receptive field by a void coefficient that is consistent with the value of the preset upsampling ratio, the semantics expressed by the selected neighborhood pixels can be enriched.

[0096] Specifically, for pixel i in the target query feature map, the server can obtain a preset hole coefficient, and based on the hole coefficient, determine the convolution processing area with the mirror pixel of the pixel i as the center pixel in the target key feature map. Since the hole coefficient is greater than 1, the receptive field of the convolution processing area is greater than the coverage area of ​​the convolution kernel in the normal convolution process, and the distance between each two adjacent pixels in the convolution processing area is determined by the hole coefficient. For example, when the hole coefficient is 2, there is 1 pixel between each two adjacent pixels. The server can use the pixel that needs to be processed by each convolution kernel in the convolution processing area as the adjacent pixel of the mirror pixel, and then perform differential convolution on the convolution processing area. In the center differential convolution process, each adjacent pixel in the convolution processing area is subtracted from the pixel i respectively, and the pixel difference obtained for each adjacent pixel is used to update the corresponding adjacent pixel to obtain the updated adjacent pixel. Then, the updated adjacent pixel is subjected to hole convolution processing to obtain the convolution processing result corresponding to the pixel i.

[0097] In the above embodiment, based on the hole coefficient, the neighborhood pixels are determined in the target key feature map. Since the hole coefficient is consistent with the upsampling ratio, the determined neighborhood pixels are more accurate. Each neighborhood pixel in the target key feature map is subjected to a central difference convolution based on the hole convolution to obtain the convolution processing result, thereby avoiding the mosaic effect during the upsampling process.

[0098] In some embodiments, the image processing method of the present application also includes: filtering noise on the high-resolution projection feature map to obtain a high-resolution denoised feature map; for pixels in the high-resolution projection feature map, determining the neighboring pixels of their mirrored pixels in the high-resolution denoised feature map, performing differential convolution on the neighboring pixels in the high-resolution denoised feature map, and obtaining the convolution processing results corresponding to the pixels in the high-resolution projection feature map.

[0099] Among them, the high-resolution denoising feature map and the high-resolution projection feature map are feature maps of the same size. Therefore, for pixel i in the high-resolution projection feature map, its mirror pixel in the high-resolution denoising feature map is the pixel at the same position as the pixel i. For example, assuming that pixel i in the high-resolution projection feature map is the pixel in the second row and the fourth column, its mirror pixel in the high-resolution denoising feature map is also the pixel in the second row and the fourth column.

[0100] Specifically, the server can also filter out noise on the high-resolution projection feature map to obtain a high-resolution denoised feature map. For pixel i in the high-resolution projection feature map, determine the mirror pixel in the obtained high-resolution denoised feature map, and then determine the neighborhood pixels of the mirror pixel in the high-resolution denoised feature map. Taking pixel i as the center pixel, differential convolution is performed on these neighborhood pixels to obtain the convolution processing result corresponding to pixel i, wherein the differential convolution is to subtract the neighborhood pixel from pixel i to update the neighborhood pixel, and then convolution processing is performed on the updated neighborhood pixel.

[0101] Furthermore, in this embodiment, based on the convolution processing results corresponding to each pixel in the target query feature map, the upsampling kernel corresponding to each pixel in the target query feature map is determined, including: based on the convolution processing results corresponding to each pixel in the target query feature map and the convolution processing results corresponding to each pixel in the high-resolution projection feature map, determining the upsampling kernel corresponding to each pixel in the target query feature map.

[0102] Specifically, for each pixel in the target query feature map, there is a corresponding pixel at the corresponding position of the high-resolution projection feature map. Therefore, when the server determines the upsampling kernel corresponding to each pixel in the target query feature map, it can combine the convolution processing result of the pixel itself and the convolution processing result of the pixel at the corresponding position of the high-resolution projection feature map to determine the upsampling kernel of the pixel.

[0103] In the above embodiment, based on the convolution processing results corresponding to each pixel in the target query feature map and the convolution processing results corresponding to each pixel in the high-resolution projection feature map, the upsampling kernel corresponding to each pixel in the target query feature map is determined, which can improve the accuracy of the obtained upsampling kernel, thereby further improving the accuracy of the upsampling result.

[0104] In some embodiments, based on the convolution processing results corresponding to each pixel in the target query feature map and the convolution processing results corresponding to each pixel in the high-resolution projection feature map, an upsampling kernel corresponding to each pixel in the target query feature map is determined, including: determining the similarity representation value corresponding to each pixel in the target query feature map based on the convolution processing results corresponding to each pixel in the target query feature map; determining the similarity representation value corresponding to each pixel in the high-resolution projection feature map based on the convolution processing results corresponding to each pixel in the high-resolution projection feature map; statistically analyzing the similarity representation values ​​of pixels at the same position in the target query feature map and the high-resolution projection feature map, and determining the upsampling kernel corresponding to each pixel in the target query feature map based on the statistical results.

[0105] Specifically, for each pixel in the target query feature map, the server can sum the convolution processing results of the pixel in each output channel, and then pass it through a ReLU activation function and 1*1 convolution to obtain the output channel K 2 Similarity representation value of 2 The similarity representation value of each channel corresponds to the K 2 The similarity corresponding to the neighborhood pixels is obtained based on the high-resolution feature map and the low-resolution feature map, which can represent the mutual similarity between the pixels in the high-resolution feature map and the pixels in the low-resolution feature map.

[0106] Similarly, for each pixel in the target high-resolution projection feature map, the server can sum the convolution processing results of the pixel in each output channel, and then pass it through a ReLU activation function and 1*1 convolution to obtain the output channel K 2 Similarity representation value of 2 The similarity representation value of each channel corresponds to the K 2 The server can calculate the similarity corresponding to the neighborhood pixels, and then for each pixel in the target query feature map, the server can perform statistics on the similarity representation value of the pixel and the similarity representation value corresponding to the pixel at the same position in the high-resolution projection feature map to obtain the statistical result of the pixel, and determine the upsampling kernel of the pixel based on the statistical result. The similarity here is obtained only based on the high-resolution feature map, which can characterize the self-similarity of the pixels of the high-resolution feature map.

[0107] Optionally, the statistics may be at least one of summation and averaging. Taking summation as an example, for pixel i in the target query feature map, the server may add the value of each channel in the pixel degree representation value of pixel i to the value of the corresponding channel in the pixel degree representation value of the pixel at the same position in the high-resolution projection feature map to obtain a statistical result.

[0108] In the above embodiment, the similarity representation values ​​of the pixels at the same position in the target query feature map and the high-resolution projection feature map are statistically analyzed, and the upsampling kernel corresponding to each pixel in the target query feature map is determined based on the statistical results. Since the upsampling kernel of each pixel combines the mutual similarity information and the self-similarity information at the same time, it can better express the semantic similarity and thus has higher accuracy.

[0109] In some embodiments, noise is filtered out of the high-resolution projection feature map to obtain a high-resolution denoised feature map, including: determining a Gaussian blur kernel; and performing Gaussian blur processing on the high-resolution projection feature map based on the Gaussian blur kernel to obtain a high-resolution denoised feature map.

[0110] Specifically, the server can sample Gaussian blur to remove local tiny noise in the high-resolution projection feature map. In the specific implementation, the server can determine the Gaussian blur kernel. The Gaussian blur kernel can be, for example, 3*3 with a variance of 1. After determining the Gaussian blur kernel, the server can perform Gaussian blur processing on the high-resolution projection feature map based on the Gaussian blur kernel to obtain a high-resolution denoised feature map.

[0111] In this embodiment, based on the Gaussian blur kernel, the high-resolution projection feature map is Gaussian blurred to obtain a high-resolution denoised feature map, which can quickly and accurately filter out noise from the high-resolution projection feature map, thereby improving the efficiency and accuracy of upsampling.

[0112] In some embodiments, for a pixel in a high-resolution projection feature map, the neighboring pixels of its mirrored pixel in a high-resolution denoising feature map are determined, and differential convolution is performed on the neighboring pixels in the high-resolution denoising feature map to obtain a convolution processing result corresponding to the pixel in the high-resolution projection feature map, including: for a pixel in the high-resolution projection feature map, determining its mirrored pixel in the high-resolution denoising feature map; based on a preset hole coefficient, determining the neighboring pixels of the mirrored pixel in the high-resolution denoising feature map, the hole coefficient is consistent with the value of the preset upsampling ratio; subtracting each neighboring pixel in the high-resolution denoising feature map from the pixel to obtain an updated neighboring pixel; performing hole convolution processing on the updated neighboring pixel to obtain a convolution processing result corresponding to the pixel in the high-resolution projection feature map.

[0113] Among them, for a pixel in the high-resolution projected feature map, its mirror pixel in the high-resolution denoised feature map is the pixel at the same position as the pixel. For example, for the pixel in the first row and second column of the high-resolution projected feature map, its mirror pixel in the high-resolution denoised feature map is the pixel in the first row and second column of the high-resolution denoised feature map. The preset upsampling ratio refers to the preset upsampling ratio for low-resolution features. The hole coefficient is consistent with the value of the preset upsampling ratio, that is, the hole coefficient is a value greater than 1.

[0114] Specifically, for any pixel j in the high-resolution projection feature map, the server determines the position identical to its pixel position in the high-resolution denoising feature map based on its pixel position in the high-resolution projection feature map, and determines the pixel at that position in the high-resolution denoising feature map as the mirror pixel of the pixel j in the high-resolution projection feature map.

[0115] After determining the mirror pixel of the pixel j, the server can obtain a preset hole coefficient, and based on the hole coefficient, determine the convolution processing area with the mirror pixel of the pixel j as the center pixel in the high-resolution denoising feature map. Since the hole coefficient is greater than 1, the receptive field of the convolution processing area is greater than the coverage area of ​​the convolution kernel in the normal convolution process, and the distance between each two adjacent pixels in the convolution processing area is determined by the hole coefficient. For example, when the hole coefficient is 2, there is 1 pixel between each two adjacent pixels. The server can use the pixel to be processed by each convolution kernel in the convolution processing area as the adjacent pixel of the mirror pixel, and then perform differential convolution on the convolution processing area. In the center differential convolution process, each adjacent pixel in the convolution processing area is respectively subtracted from the pixel j, and the pixel difference obtained by subtracting each adjacent pixel is used to update the corresponding adjacent pixel to obtain the updated adjacent pixel. Then, the hole convolution process is performed on the updated adjacent pixel to obtain the convolution processing result corresponding to the pixel j.

[0116] In the above embodiment, based on the hole coefficient, the neighborhood pixels are determined in the high-resolution denoising feature map. Since the hole coefficient is consistent with the upsampling ratio, the determined neighborhood pixels are more accurate. Each neighborhood pixel in the high-resolution denoising feature map is subjected to central difference convolution based on the hole convolution to obtain the convolution processing result, thereby avoiding the mosaic effect in the upsampling process.

[0117] In some embodiments, based on the upsampling kernel corresponding to each pixel in the target query feature map, the low-resolution feature map is upsampled to obtain an upsampled feature map, including: performing bilinear interpolation on the low-resolution feature map according to the resolution of the high-resolution projection feature map to obtain an intermediate feature map; based on the upsampling kernel corresponding to each pixel in the target query feature map, the intermediate feature map is upsampled to obtain an upsampled feature map.

[0118] Taking into account that in the process of determining the upsampling kernel, the low-resolution projection feature map is upsampled according to the resolution of the high-resolution projection feature map, the determined neighborhood pixels are located based on the size of the high-resolution feature map. In the process of upsampling according to the upsampling kernel, the neighborhood pixels at the same position as in the process of determining the upsampling kernel can be determined, thereby further improving the accuracy of upsampling. Based on this, the server can perform bilinear interpolation on the low-resolution feature map according to the resolution of the high-resolution feature map to obtain the resolution and high-resolution feature map. Figure 1 The intermediate feature map can be upsampled based on the upsampling kernel corresponding to each pixel in the target query feature map to obtain the upsampled feature map.

[0119] In the above embodiment, bilinear interpolation is performed on the low-resolution feature map according to the resolution of the high-resolution projection feature map to obtain an intermediate feature map, and the intermediate feature map is upsampled based on the upsampling kernel corresponding to each pixel in the target query feature map to obtain an upsampled feature map, thereby further improving the accuracy of the upsampling result.

[0120] In some embodiments, based on the upsampling kernel corresponding to each pixel in the target query feature map, the intermediate feature map is upsampled to obtain an upsampled feature map, including: for a pixel in the target query feature map, determining the target neighborhood pixel corresponding to the pixel in the intermediate feature map, the target neighborhood pixel and the neighborhood pixel position of the pixel in the target key feature map are the same; based on the upsampling kernel corresponding to the pixel, the target neighborhood pixel is weighted summed to obtain the upsampled pixel corresponding to the pixel; the upsampled pixel corresponding to each pixel in the target query feature map is used to form an upsampled feature map.

[0121] Specifically, the server can determine the target neighborhood pixels corresponding to each pixel in the target query feature map in the intermediate feature map, use the upsampling kernel corresponding to each pixel as a weighting coefficient, and perform weighted summation of the target neighborhood pixels corresponding to each pixel through upsampling of each pixel, thereby obtaining the upsampled pixels corresponding to each pixel in the target query feature map. For example, assuming that the upsampling kernel corresponding to a pixel in the target query feature map is [x1, x2, x3, ..., x8], and the pixel values ​​of the target neighborhood pixels corresponding to the upsampling kernel are a1, a2, a3, ..., a8, respectively, then the final pixel value of the upsampled pixel of the pixel is x1a1+x2a2+x3a3+x4a4+ ... +x8a.

[0122] Since the target query feature map is obtained based on the high-resolution feature map, the pixels in the target query feature map correspond one-to-one to the pixels in the high-resolution feature map. Therefore, the upsampled pixels corresponding to each pixel in the target query feature map are the upsampled pixels corresponding to the pixels at the corresponding position in the high-resolution feature map. The image composed of these upsampled pixels is the upsampling result obtained by upsampling the low-resolution feature map, that is, the upsampled feature map corresponding to the low-resolution feature map.

[0123] It can be understood that in this embodiment, for a pixel in the target query feature map, when determining its corresponding target neighborhood pixel in the intermediate feature map, the determined target neighborhood pixel is the same as the neighborhood pixel position of the pixel in the target query feature map in the target key feature map, and the upsampling kernel coefficient can represent the similarity between the pixel and its neighborhood pixel in the target key feature map. Therefore, when the pixels at the same position in the intermediate feature map are weighted by the upsampling kernel value, the accuracy of finding that detail in the sampling feature map can be ensured. Optionally, if in the process of determining the upsampling kernel, the neighborhood pixels are selected based on the hole coefficient, then when selecting the target neighborhood pixels, the neighborhood pixels can be determined based on the same hole coefficient, and the target neighborhood pixels selected in the upsampling process are determined to have the same position as the neighborhood pixels selected in the upsampling kernel determination process, thereby better ensuring the accuracy of the upsampling result.

[0124] In some embodiments, obtaining a low-resolution feature map and a high-resolution feature map extracted from the same image includes: obtaining an image to be segmented, extracting shallow features from the image to be segmented, and obtaining a high-resolution feature map; inputting the image into an image segmentation network to obtain a low-resolution feature map.

[0125] Specifically, in this embodiment, the image processing method of the present application can be applied to image segmentation scenarios. When performing image segmentation, the server can obtain the image to be segmented, extract shallow features of the image to be segmented, obtain a high-resolution feature map, and input the image to be segmented into the image segmentation network to obtain a low-resolution feature map. The image segmentation result can then be determined by using the high-resolution feature map as a guiding feature map to guide the upsampling process of the low-resolution feature map to obtain an upsampling feature map.

[0126] refer to Figure 4, is a schematic diagram of the overall process of the image processing method of the present application when applying image segmentation in some embodiments, wherein the main network is the representative deep network Segmenter, and the "shallow feature extraction" operation adopts a Conv3×3-GroupNorm-ReLU-Conv3×3 convolution block. Specifically, the image to be segmented undergoes a 3x3 convolution to obtain a feature with 64 channels. After group normalization (Group Normalization) and ReLU activation function, it undergoes another 3x3 convolution to obtain a feature with a channel size (channel size) of 32, which is used as the HR guided feature (i.e., high-resolution feature map). After the image passes through the Segmenter main network, the LR feature (i.e., low-resolution feature map) is obtained. After being guided by the HR guided feature, the LR feature is upsampled 4 times by the image processing method of the present application to obtain the HR feature. Figure 4 The final "segmentation head" in is a 1x1 convolution.

[0127] In the above embodiment, since upsampling is performed using the image processing method of the present application during the image segmentation process, a better upsampling effect can be obtained, thereby improving the performance of the image segmentation task and making the obtained segmentation result more accurate.

[0128] In a specific embodiment, Figure 5 As shown, an image processing method is provided, which is executed by a computer device, which may be Figure 1 In the server 104 or terminal 103, in the embodiment of the present application, the method is applied to Figure 1 Taking the server 104 in the example as an example, the following steps are included:

[0129] Step 502: Obtain a low-resolution feature map and a high-resolution feature map extracted from the same image. The resolution of the high-resolution feature map is higher than that of the low-resolution feature map.

[0130] Among them, the low-resolution feature map refers to a feature map with lower resolution, and the high-resolution feature map refers to a feature map with higher resolution. The resolution of the high-resolution feature map is higher than that of the low-resolution feature map. The low-resolution feature map and the high-resolution feature map are feature maps extracted from the same image. The low-resolution feature map is the feature map to be upsampled, and the high-resolution feature map is used to provide guiding feature information during the upsampling process of the feature map to be upsampled, so as to guide the upsampling process of the feature map to be upsampled.

[0131] Step 504: linearly project the high-resolution feature map to obtain a high-resolution projection feature map, and linearly project the low-resolution feature map to obtain a low-resolution projection feature map.

[0132] For example, the server may perform linear projection on the high-resolution feature map and the low-resolution feature map respectively through group normalization and a 1*1 convolution layer.

[0133] Step 506 : Upsample the low-resolution projected feature map according to the resolution of the high-resolution projected feature map to obtain a target key feature map.

[0134] Exemplarily, the server may perform bilinear interpolation upsampling on the low-resolution projection feature map according to the resolution of the high-resolution projection feature map to obtain a target key feature map.

[0135] In step 508 , the target key feature map is used as the input map, and the high-resolution projection feature map is used as the guide map. Guided filtering is performed based on the guide map and the input map to obtain the target query feature map.

[0136] Among them, guided filtering is an image filtering technology that filters the initial image p (input image) through a guiding image I, so that the final output image is roughly similar to the initial image P, but the texture part is similar to the guiding image I.

[0137] Step 510: for a pixel in the target query feature map, determine a mirror pixel with the same position as the pixel in the target key feature map, and determine the neighboring pixels of the mirror pixel in the target key feature map based on a preset hole coefficient.

[0138] The preset hole coefficient is consistent with the preset upsampling ratio. The preset upsampling ratio refers to the preset upsampling ratio for low-resolution features. The hole coefficient is consistent with the preset upsampling ratio, that is, the hole coefficient is a value greater than 1.

[0139] In step 512, each neighborhood pixel in the target key feature map is subtracted from the pixel in the target query feature map to obtain an updated neighborhood pixel, and a dilated convolution process is performed on the updated neighborhood pixel to obtain a convolution process result corresponding to the pixel.

[0140] Specifically, for pixel i in the target query feature map, the server can determine the convolution processing area with the mirror pixel of pixel i as the center pixel in the target key feature map based on the hole coefficient, and use the pixels that each convolution kernel needs to process in the convolution processing area as the adjacent pixels of the mirror pixel, and then perform differential convolution on the convolution processing area. In the center differential convolution process, each adjacent pixel in the convolution processing area is subtracted from the pixel i respectively, and the pixel difference obtained for each adjacent pixel is used to update the corresponding adjacent pixel to obtain the updated adjacent pixel, and then perform hole convolution processing on the updated adjacent pixel to obtain the convolution processing result corresponding to the pixel i.

[0141] In step 514 , based on the Gaussian blur kernel, Gaussian blur processing is performed on the high-resolution projection feature map to obtain a high-resolution denoised feature map.

[0142] Specifically, the server can sample Gaussian blur to remove local tiny noise in the high-resolution projection feature map. In the specific implementation, the server can determine the Gaussian blur kernel. The Gaussian blur kernel can be, for example, 3*3 with a variance of 1. After determining the Gaussian blur kernel, the server can perform Gaussian blur processing on the high-resolution projection feature map based on the Gaussian blur kernel to obtain a high-resolution denoised feature map.

[0143] Step 516: for a pixel in the high-resolution projection feature map, determine its mirror image pixel in the high-resolution denoising feature map, and determine the neighboring pixels of the mirror image pixel in the high-resolution denoising feature map based on a preset hole coefficient.

[0144] The hole coefficient is consistent with the value of the preset upsampling ratio. For a pixel in the high-resolution projection feature map, its mirror pixel in the high-resolution denoising feature map is the pixel at the same position as the pixel.

[0145] In step 518, each neighborhood pixel in the high-resolution denoising feature map is subtracted from the pixel in the high-resolution projection feature map to obtain an updated neighborhood pixel, and a dilated convolution process is performed on the updated neighborhood pixel to obtain a convolution process result corresponding to the pixel in the high-resolution projection feature map.

[0146] Specifically, for any pixel j in the high-resolution projection feature map, the server determines the position identical to its pixel position in the high-resolution denoising feature map based on its pixel position in the high-resolution projection feature map, and determines the pixel at that position in the high-resolution denoising feature map as the mirror pixel of the pixel j in the high-resolution projection feature map.

[0147] After determining the mirror pixel of the pixel j, the server can obtain a preset hole coefficient, and based on the hole coefficient, determine the convolution processing area with the mirror pixel of the pixel j as the center pixel in the high-resolution denoising feature map, and use the pixels that need to be processed by each convolution kernel in the convolution processing area as the adjacent pixels of the mirror pixel. Then, differential convolution is performed on the convolution processing area. In the center differential convolution process, each adjacent pixel in the convolution processing area is respectively subtracted from the pixel j, and the pixel difference value obtained by subtracting each adjacent pixel is used to update the corresponding adjacent pixel to obtain the updated adjacent pixel. Then, hole convolution processing is performed on the updated adjacent pixel to obtain the convolution processing result corresponding to the pixel j.

[0148] Step 520 : Determine a similarity representation value corresponding to each pixel in the target query feature map based on the convolution processing result corresponding to each pixel in the target query feature map.

[0149] Step 522 : Determine a similarity representation value corresponding to each pixel in the high-resolution projection feature map based on the convolution processing result corresponding to each pixel in the high-resolution projection feature map.

[0150] Specifically, for each pixel in the target query feature map, the server can sum the convolution processing results of the pixel in each output channel, and then pass it through a ReLU activation function and 1*1 convolution to obtain the output channel K 2 Similarity representation value of 2 The similarity representation value of each channel corresponds to the K 2 The similarity corresponding to the neighboring pixels.

[0151] Similarly, for each pixel in the target high-resolution projection feature map, the server can sum the convolution processing results of the pixel in each output channel, and then pass it through a ReLU activation function and 1*1 convolution to obtain the output channel K 2 Similarity representation value of 2 The similarity representation value of each channel corresponds to the K 2 The similarity corresponding to the neighboring pixels.

[0152] In step 524 , the similarity representation values ​​of the pixels at the same position in the target query feature map and the high-resolution projection feature map are statistically analyzed, and an upsampling kernel corresponding to each pixel in the target query feature map is determined based on the statistical results.

[0153] Specifically, for each pixel in the target query feature map, the server can perform statistics on the similarity representation value of the pixel and the similarity representation value corresponding to the pixel at the same position in the high-resolution projection feature map to obtain the statistical result of the pixel, and determine the upsampling kernel of the pixel based on the statistical result.

[0154] In step 526 , bilinear interpolation is performed on the low-resolution feature map according to the resolution of the high-resolution projection feature map to obtain an intermediate feature map.

[0155] Taking into account that in the process of determining the upsampling kernel, the low-resolution projection feature map is upsampled according to the resolution of the high-resolution projection feature map, the determined neighborhood pixels are located based on the size of the high-resolution feature map. In the process of upsampling according to the upsampling kernel, the neighborhood pixels at the same position as in the process of determining the upsampling kernel can be determined, thereby further improving the accuracy of upsampling. Based on this, the server can perform bilinear interpolation on the low-resolution feature map according to the resolution of the high-resolution feature map to obtain the resolution and high-resolution feature map. Figure 1 The intermediate feature map.

[0156] Step 528: For a pixel in the target query feature map, determine the target neighborhood pixels corresponding to the pixel in the intermediate feature map, where the target neighborhood pixels have the same position as the neighborhood pixels of the pixel in the target key feature map.

[0157] Step 530: Based on the upsampling kernel corresponding to the pixel, weighted summation is performed on the target neighborhood pixels to obtain the upsampling pixel corresponding to the pixel; the upsampling pixel corresponding to each pixel in the target query feature map is used to form the upsampling feature map.

[0158] The server can determine the target neighborhood pixels corresponding to each pixel in the target query feature map in the intermediate feature map, use the upsampling kernel corresponding to each pixel as the weighting coefficient, and perform weighted summation on the target neighborhood pixels corresponding to each pixel through upsampling of each pixel, so as to obtain the upsampled pixels corresponding to each pixel in the target query feature map.

[0159] In the above embodiment, a high-resolution feature map is used to help restore details in the upsampling process of the low-resolution feature map, the high-resolution feature map is controllably aligned with the low-resolution feature map, and the similarity is calculated through differential convolution, which can flexibly capture the true local similarity and avoid the mosaic effect.

[0160] In some embodiments, the present application further provides an application scenario, in which the image processing method of the present application is applied to an image segmentation task, referring to Figure 4 The solution of this application can be used to upsample the low-resolution feature map (hereinafter referred to as LR feature) output by the main network of image segmentation in order to obtain a more refined segmentation result. It involves two parts of input, one is the LR feature to be upsampled, and the other is the HR guided feature (i.e., high-resolution feature map), and the output is the upsampled HR feature. The following is a detailed introduction:

[0161] 1. Establish training data set and data preprocessing

[0162] For image segmentation tasks, you can choose the ADE20K dataset and use the corresponding data processing methods in the mmsegmentation code library for data preprocessing.

[0163] It is understandable that the solution of the present application is not limited to image segmentation tasks, but can be applied to any application tasks involving feature upsampling, and the corresponding training data set can be selected for actual applications.

[0164] 2. Deep Neural Network Architecture

[0165] In this embodiment, the representative deep network Segmenter of semantic segmentation task can be used as the main network. Figure 4 The "shallow feature extraction" operation in

[15] can use a Conv3×3-GroupNorm-ReLU-Conv3×3 convolution block. Specifically, the original image undergoes a 3x3 convolution to obtain a feature with 64 channels. After group normalization (GroupNormalization) and ReLU activation function, it undergoes another 3x3 convolution to obtain a feature with a channel size of 32, which serves as the HR guide feature. After the image passes through the Segmenter main network, the LR features are obtained. After being guided by the HR guide features, the solution of this application is used to perform a 4x upsampling to obtain the HR features.

[0166] 3. Feature upsampling model and design

[0167] refer to Figure 6 , is the overall framework diagram of upsampling using the image processing method of the present application in this embodiment, refer to Figure 6 , given an LR feature x∈R hw×C , and an HR guided feature y∈R HW×c , we hope to get HR features after upsampling x Specifically, for each HR pixel i∈{1,…,HW}, the pixels in its adjacent LR features are usually weighted according to the upsampling kernel to estimate the HR feature element

[0168]

[0169] Where N(i) is the neighboring pixels of pixel i, is the number of neighborhood pixels, K is the upsampling kernel size (kernel size), and in the specific implementation, K=3 can be set. It's K 2The similarity score calculated by the neighborhood pixels is obtained by the Softmax operation. i ) is used as the upsampling kernel at position i.

[0170] To calculate the similarity score s i , we can first perform a linear projection (GroupNormalization+1x1 convolution) on y and x to get q∈R HW×D and k∈R hw×D , their channel numbers are all D, and D=32 can be set in the specific implementation.

[0171] Continue to refer Figure 6 , we can use guided filtering (GF) to transform q and controllably project it into the semantic space where k is located to perform semantic-aware mutual-alignment. Specifically, we can first upsample the k bilinear difference to obtain Then use the guided filtering code toolkit to convert q and They are used as the guide map and input map in the guided filtering, and the output filtering result is q GF ∈R HW×D Therefore, the semantically perceived mutual similarity score of pixel i is:

[0172]

[0173] In addition to the mutual similarity scores obtained by feature alignment in the semantic space, detail-aware self-alignment can also be performed, that is, the self-alignment of q itself is used to help preserve the fidelity of detail information. Figure 6 , Gaussian blur can be used to remove the local small noise of q as the target feature of the selected neighborhood, where f∈R 3×3 is a 3×3 Gaussian blur kernel with a variance of 1. We can thus calculate the self-similarity score (s d ) i for:

[0174]

[0175] Combining formulas (2) and (3), the final total similarity score is

[0176] s i =(s s ) i +(s d ) i (4)

[0177] Continue to refer Figure 6 In the embodiment of the present application, a paired central difference convolution (PCDC) module (PCDC-Block) is designed to implement the similarity calculation function sim(·,·) in formulas (2) and (3). The input target query feature map Query and the target key feature map Key are q in formula (2) respectively. GG and Or q and q in formula (3) respectively blur .

[0178] refer to Figure 7 , the framework diagram of PCDC-Block can be referred to Figure 7 As shown. First, Query and Key pass through a group normalization layer (Group Normalization), and they share affine parameters (shared Affine Weights). Normalization is respectively obtained and They are input into the PCDC layer and the output is v∈R HW×G In the specific implementation, the number of output channels G can be set to 128. For the i-th pixel, the output v of the g-th dimension is ig , the calculation process of PCDC is:

[0179]

[0180] in, is the convolution weight, b∈R G is the convolution bias, both are learnable parameters. d is the dth channel of the input, d is less than or equal to D. After obtaining v, after a ReLU activation function and a 1x1 convolution, the output channel is K 2 The fraction of , as s in formula (2) s or s in (3) d , where K for each pixel 2 The value of the channel corresponds to the K 2 The similarity corresponding to the neighboring pixels.

[0181] In the implementation of PCDC, a dilated convolution with a dilated coefficient equal to the upsampling ratio can be used to implement a refined neighborhood pixel selection function on the HR feature. Therefore, the above formula (5) can be understood as The i-th pixel in is taken as the difference center, The neighborhood pixels in the image are subjected to differential convolution based on hole convolution, which can better eliminate the mosaic effect.

[0182] In summary, in the embodiment of the present application, the similarity calculation function of PCDC-Block and the refined neighborhood pixel selection are used. Calculate the similarity in formula (4) and bring it into formula (1) to get the final upsampling result

[0183] 4. Model training

[0184] For different network architectures, the default training settings provided in the mmsegmentation code library (including optimization scheme, learning rate, loss function, number of training iterations, etc.) were used for training. After replacing the default bilinear interpolation upsampling method used in the network with a guided feature upsampling scheme based on controllable feature alignment, other training settings remained unchanged.

[0185] 5. Implementation logic and processing flow:

[0186] refer to Figure 8 , which is a schematic diagram of the implementation logic of the image segmentation task in this embodiment, refer to Figure 8 In the image segmentation task, the front-end A receives the image data and then uploads it to the back-end. The back-end uses the deep network containing this technical solution to segment the image and then outputs it to the front-end A.

[0187] In this embodiment, the image processing method of the present application has at least the following beneficial effects:

[0188] 1) A better visual effect of feature upsampling can be obtained. Figure 9 As shown in the figure, it is a schematic diagram of the visual effect comparison of image semantic segmentation. The first row is the up-sampled features, and the second row is the segmentation results. Figure 9 It can be seen that when different upsampling methods are applied to the Segmenter network for semantic segmentation, the upsampling feature results obtained by the solution of the present application have clear boundaries and do not have the mosaic effect in comparison method 1 and comparison method 2.

[0189] 2) With the help of better feature alignment and semantic and flexible similarity calculation, the trained deep network can achieve better segmentation performance. Figure 10 As shown in the figure, on the image semantic segmentation task, using 5 different networks, the performance comparison of various upsampling schemes on the ADE20K dataset, where mIoU is the mean intersection over union index, mAcc is the pixel average accuracy index, and bIoU is the boundary intersection over union index. Figure 10 It can be seen that the solutions of this application can achieve leading performance.

[0190] In some embodiments, the present application also provides another application scenario, in which reference is made to Figure 11 The image processing method of the present application is applied to the target detection task. Specifically, the server can obtain the image to be detected, perform shallow feature extraction on the image, obtain a high-resolution feature map, and input the image to be detected into the target detection main network to obtain a low-resolution feature map, perform linear projection on the high-resolution feature map, obtain a high-resolution projection feature map, perform linear projection on the low-resolution feature map, obtain a low-resolution projection feature map, perform semantic-aware mutual alignment based on the high-resolution projection feature map and the low-resolution projection feature map, and calculate the mutual similarity score, perform detail-aware self-alignment based on the high-resolution projection feature map, and calculate the self-similarity score, finally combine the mutual similarity score and the self-similarity score to obtain an upsampling kernel, and then upsample the low-resolution feature map based on the upsampling kernel to obtain an upsampled feature map. After obtaining the upsampled feature map, the server can determine the target detection result based on the upsampled feature map. The specific implementation process can be referred to in the above embodiment, and this application will not go into details here.

[0191] It should be understood that, although the steps in the flowcharts of the above embodiments are shown in sequence as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts of the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily to be performed in sequence, but can be performed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0192] Based on the same inventive concept, embodiments of the present application also provide an image processing device for implementing the aforementioned image processing method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following image processing device embodiments can be found in the above-described limitations on the image processing method and will not be further elaborated here.

[0193] In some embodiments, as Figure 12 As shown, an image processing device 1200 is provided, including:

[0194] A feature map acquisition module 1202 is configured to acquire a low-resolution feature map and a high-resolution feature map respectively extracted from the same image, wherein the high-resolution feature map has a higher resolution than the low-resolution feature map;

[0195] A feature map projection module 1204 is configured to perform linear projection on the high-resolution feature map to obtain a high-resolution projection feature map, and to perform linear projection on the low-resolution feature map to obtain a low-resolution projection feature map;

[0196] A query feature map determining module 1206 is configured to upsample the low-resolution projected feature map according to the resolution of the high-resolution projected feature map to obtain a target key feature map, and determine a target query feature map based on the high-resolution projected feature map;

[0197] The differential convolution module 1208 is used to determine, for a pixel in the target query feature map, a mirrored pixel at the same position as the pixel in the target key feature map, determine the neighboring pixels of the mirrored pixel, perform differential convolution on the neighboring pixels, and obtain a convolution processing result corresponding to the pixel;

[0198] An upsampling kernel determining module 1210 is configured to determine an upsampling kernel corresponding to each pixel in the target query feature map based on a convolution processing result corresponding to each pixel in the target query feature map;

[0199] The feature map upsampling module 1212 is configured to upsample the low-resolution feature map based on the upsampling kernel corresponding to each pixel in the target query feature map to obtain an upsampled feature map.

[0200] The image processing device obtains a low-resolution feature map and a high-resolution feature map extracted from the same image, performs linear projection on the high-resolution feature map to obtain a high-resolution projection feature map, performs linear projection on the low-resolution feature map to obtain a low-resolution projection feature map, upsamples the low-resolution projection feature map according to the resolution of the high-resolution projection feature map to obtain a target key feature map, determines a target query feature map based on the high-resolution projection feature map, determines a mirrored pixel with the same position as the pixel in the target query feature map in the target key feature map, determines the neighboring pixels of the mirrored pixel, performs differential convolution on the neighboring pixels, and obtains the pixel corresponding to the target key feature map. The convolution processing result is based on the convolution processing result corresponding to each pixel in the target query feature map, and the upsampling kernel corresponding to each pixel in the target query feature map is determined. Based on the upsampling kernel corresponding to each pixel in the target query feature map, the low-resolution feature map is upsampled to obtain the upsampled feature map, so that the high-resolution feature map can be used to guide the upsampling process of the low-resolution feature map, which can help restore the details in the upsampling process of the low-resolution feature map, making the feature map obtained by upsampling more accurate, and the upsampling kernel can be determined by differential convolution, which can flexibly capture the true local similarity, further improving the accuracy of the feature map obtained by upsampling.

[0201] In some embodiments, the query feature map determination module is also used to project the high-resolution projection feature map into the semantic space where the low-resolution projection feature map is located to obtain a high-resolution aligned feature map that is aligned with the low-resolution projection feature map in the semantic space; and determine the high-resolution aligned feature map as the target query feature map.

[0202] In some embodiments, the query feature map determination module is also used to use the target key feature map as the input map and the high-resolution projection feature map as the guide map; perform guided filtering based on the guide map and the input map to obtain a high-resolution aligned feature map that is aligned with the low-resolution projection feature map in the semantic space.

[0203] In some embodiments, the differential convolution module is also used to obtain a preset hole coefficient, which is consistent with the value of the preset upsampling ratio; based on the hole coefficient, the neighborhood pixels of the mirrored pixel are determined in the target key feature map; each neighborhood pixel in the target key feature map is subtracted from the pixel to obtain an updated neighborhood pixel; and the updated neighborhood pixel is subjected to hole convolution processing to obtain the convolution processing result corresponding to the pixel.

[0204] In some embodiments, the image processing method of the present application further includes: a feature map denoising module for filtering noise from the high-resolution projection feature map to obtain a high-resolution denoised feature map; for a pixel in the high-resolution projection feature map, determining the neighboring pixels of its mirrored pixel in the high-resolution denoised feature map, performing differential convolution on the neighboring pixels in the high-resolution denoised feature map, and obtaining a convolution processing result corresponding to the pixel in the high-resolution projection feature map; and an upsampling kernel determination module for:

[0205] Based on the convolution processing results corresponding to each pixel in the target query feature map and the convolution processing results corresponding to each pixel in the high-resolution projection feature map, an upsampling kernel corresponding to each pixel in the target query feature map is determined.

[0206] In some embodiments, the upsampling kernel determination module is further used to: determine the similarity representation value corresponding to each pixel in the target query feature map based on the convolution processing result corresponding to each pixel in the target query feature map; determine the similarity representation value corresponding to each pixel in the high-resolution projection feature map based on the convolution processing result corresponding to each pixel in the high-resolution projection feature map; perform statistics on the similarity representation values ​​of pixels at the same position in the target query feature map and the high-resolution projection feature map, and determine the upsampling kernel corresponding to each pixel in the target query feature map based on the statistical results.

[0207] In some embodiments, the feature map denoising module is further used to determine a Gaussian blur kernel; based on the Gaussian blur kernel, Gaussian blur processing is performed on the high-resolution projection feature map to obtain a high-resolution denoised feature map.

[0208] In some embodiments, the feature map denoising module is further used to: determine, for a pixel in a high-resolution projection feature map, its mirror pixel in a high-resolution denoising feature map; determine, based on a preset hole coefficient, the neighborhood pixels of the mirror pixel in the high-resolution denoising feature map, where the hole coefficient is consistent with the value of a preset upsampling ratio; subtract each neighborhood pixel in the high-resolution denoising feature map from the pixel to obtain an updated neighborhood pixel; perform hole convolution processing on the updated neighborhood pixel to obtain a convolution processing result corresponding to the pixel in the high-resolution projection feature map.

[0209] In some embodiments, the feature map upsampling module is further used to: perform bilinear interpolation on the low-resolution feature map according to the resolution of the high-resolution projection feature map to obtain an intermediate feature map; and upsample the intermediate feature map based on the upsampling kernel corresponding to each pixel in the target query feature map to obtain an upsampled feature map.

[0210] In some embodiments, the feature map upsampling module is further used to: for pixels in the target query feature map, determine the target neighborhood pixels corresponding to the pixels in the intermediate feature map, and the target neighborhood pixels have the same position as the neighborhood pixels of the pixels in the target key feature map; based on the upsampling kernel corresponding to the pixels, perform weighted summation on the target neighborhood pixels to obtain the upsampling pixels corresponding to the pixels; the upsampling pixels corresponding to each pixel in the target query feature map are used to form an upsampling feature map.

[0211] In some embodiments, the feature map acquisition module is also used to obtain the image to be segmented, extract shallow features from the image to be segmented, and obtain a high-resolution feature map; and input the image to be segmented into the image segmentation network to obtain a low-resolution feature map.

[0212] Each module in the above-mentioned image processing device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0213] In some embodiments, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 13As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device can be used to store image data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image processing method is implemented.

[0214] In some embodiments, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 14 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image processing method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0215] Those skilled in the art will understand that Figure 13 、 Figure 14The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0216] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above-mentioned image processing method when executing the computer program.

[0217] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned image processing method are implemented.

[0218] In some embodiments, a computer program product is provided, comprising a computer program, which implements the steps of the above-mentioned image processing method when executed by a processor.

[0219] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0220] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0221] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0222] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An image processing method, characterized in that: The method comprises: Obtaining a low-resolution feature map and a high-resolution feature map respectively extracted from the same image, wherein the resolution of the high-resolution feature map is higher than that of the low-resolution feature map; Performing linear projection on the high-resolution feature map to obtain a high-resolution projection feature map, and performing linear projection on the low-resolution feature map to obtain a low-resolution projection feature map; Upsampling the low-resolution projected feature map according to the resolution of the high-resolution projected feature map to obtain a target key feature map, and determining a target query feature map based on the high-resolution projected feature map; For a pixel in the target query feature map, determine a mirrored pixel at the same position as the pixel in the target key feature map, determine neighboring pixels of the mirrored pixel, perform differential convolution on the neighboring pixels, and obtain a convolution processing result corresponding to the pixel; Determine an upsampling kernel corresponding to each pixel in the target query feature map based on a convolution processing result corresponding to each pixel in the target query feature map; Based on the upsampling kernel corresponding to each pixel in the target query feature map, the low-resolution feature map is upsampled to obtain an upsampled feature map.

2. The method according to claim 1, characterized in that The determining of the target query feature map based on the high-resolution projection feature map includes: Projecting the high-resolution projected feature map into the semantic space where the low-resolution projected feature map is located to obtain a high-resolution aligned feature map that is aligned with the low-resolution projected feature map in the semantic space; The high-resolution aligned feature map is determined as a target query feature map.

3. The method according to claim 2, characterized in that The step of projecting the high-resolution projection feature map to the semantic space where the low-resolution projection feature map is located to obtain a high-resolution aligned feature map aligned with the low-resolution projection feature map in the semantic space includes: Using the target key feature map as an input map and the high-resolution projected feature map as a guide map; Guided filtering is performed based on the guide map and the input map to obtain a high-resolution aligned feature map that is aligned with the low-resolution projected feature map in the semantic space.

4. The method according to claim 1, wherein The determining of the neighboring pixels of the mirrored pixel, performing differential convolution on the neighboring pixels, and obtaining the convolution processing result corresponding to the pixel includes: Obtaining a preset hole coefficient, where the hole coefficient is consistent with a value of a preset upsampling ratio; Determining, in the target key feature map, neighboring pixels of the mirrored pixel based on the hole coefficient; Subtract each neighboring pixel in the target key feature map from the pixel to obtain an updated neighboring pixel; Performing a dilated convolution process on the updated neighborhood pixels to obtain a convolution process result corresponding to the pixels.

5. The method according to claim 1, characterized in that The method further comprises: Performing noise filtering on the high-resolution projection feature map to obtain a high-resolution denoised feature map; For a pixel in the high-resolution projection feature map, determine a neighboring pixel of its mirrored pixel in the high-resolution denoising feature map, perform differential convolution on the neighboring pixel in the high-resolution denoising feature map, and obtain a convolution processing result corresponding to the pixel in the high-resolution projection feature map; The determining, based on the convolution processing result corresponding to each pixel in the target query feature map, an upsampling kernel corresponding to each pixel in the target query feature map includes: An upsampling kernel corresponding to each pixel in the target query feature map is determined based on a convolution processing result corresponding to each pixel in the target query feature map and a convolution processing result corresponding to each pixel in the high-resolution projection feature map.

6. The method according to claim 5, characterized in that The determining, based on the convolution processing result corresponding to each pixel in the target query feature map and the convolution processing result corresponding to each pixel in the high-resolution projection feature map, an upsampling kernel corresponding to each pixel in the target query feature map includes: Determining a similarity representation value corresponding to each pixel in the target query feature map based on a convolution processing result corresponding to each pixel in the target query feature map; Determining a similarity representation value corresponding to each pixel in the high-resolution projection feature map based on a convolution processing result corresponding to each pixel in the high-resolution projection feature map; The similarity representation values ​​of pixels at the same position in the target query feature map and the high-resolution projection feature map are statistically analyzed, and an upsampling kernel corresponding to each pixel in the target query feature map is determined based on the statistical results.

7. The method according to claim 5, characterized in that The step of filtering the high-resolution projection feature map to obtain a high-resolution denoised feature map includes: Determine the Gaussian blur kernel; Based on the Gaussian blur kernel, Gaussian blur processing is performed on the high-resolution projection feature map to obtain a high-resolution denoised feature map.

8. The method according to claim 5, characterized in that The step of determining, for a pixel in the high-resolution projection feature map, a neighboring pixel of a mirrored pixel in the high-resolution denoising feature map, performing differential convolution on the neighboring pixel in the high-resolution denoising feature map, and obtaining a convolution processing result corresponding to the pixel in the high-resolution projection feature map includes: For a pixel in the high-resolution projection feature map, determining its mirror image pixel in the high-resolution denoising feature map; Determining, in the high-resolution denoising feature map, neighboring pixels of the mirrored pixel based on a preset hole coefficient, wherein the hole coefficient is consistent with a value of a preset upsampling ratio; Subtract each neighborhood pixel in the high-resolution denoising feature map from the pixel to obtain an updated neighborhood pixel; Performing dilated convolution processing on the updated neighborhood pixels to obtain convolution processing results corresponding to the pixels in the high-resolution projection feature map.

9. The method according to claim 1, characterized in that The upsampling of the low-resolution feature map based on the upsampling kernel corresponding to each pixel in the target query feature map to obtain the upsampled feature map includes: Performing bilinear interpolation on the low-resolution feature map according to the resolution of the high-resolution projection feature map to obtain an intermediate feature map; Based on the upsampling kernel corresponding to each pixel in the target query feature map, the intermediate feature map is upsampled to obtain an upsampled feature map.

10. The method according to claim 9, characterized in that The upsampling of the intermediate feature map based on the upsampling kernel corresponding to each pixel in the target query feature map to obtain the upsampled feature map includes: For a pixel in the target query feature map, determining a target neighborhood pixel corresponding to the pixel in the intermediate feature map, wherein the target neighborhood pixel has the same position as the neighborhood pixel of the pixel in the target key feature map; Based on the upsampling kernel corresponding to the pixel, performing weighted summation on the target neighborhood pixels to obtain the upsampling pixel corresponding to the pixel; The upsampled pixels corresponding to each pixel in the target query feature map are used to form an upsampled feature map.

11. The method according to any one of claims 1 to 10, characterized in that The obtaining of the low-resolution feature map and the high-resolution feature map respectively extracted from the same image includes: Acquire an image to be segmented, extract shallow features from the image to be segmented, and obtain a high-resolution feature map; The image to be segmented is input into an image segmentation network to obtain a low-resolution feature map.

12. An image processing device, characterized in that: The device comprises: A feature map acquisition module, configured to acquire a low-resolution feature map and a high-resolution feature map respectively extracted from the same image, wherein the resolution of the high-resolution feature map is higher than that of the low-resolution feature map; A feature map projection module is used to perform linear projection on the high-resolution feature map to obtain a high-resolution projection feature map, and to perform linear projection on the low-resolution feature map to obtain a low-resolution projection feature map; a query feature map determining module, configured to upsample the low-resolution projected feature map according to the resolution of the high-resolution projected feature map to obtain a target key feature map, and determine a target query feature map based on the high-resolution projected feature map; A differential convolution module is configured to determine, for a pixel in the target query feature map, a mirrored pixel at the same position as the pixel in the target key feature map, determine neighboring pixels of the mirrored pixel, perform differential convolution on the neighboring pixels, and obtain a convolution processing result corresponding to the pixel; An upsampling kernel determining module, configured to determine an upsampling kernel corresponding to each pixel in the target query feature map based on a convolution processing result corresponding to each pixel in the target query feature map; The feature map upsampling module is used to upsample the low-resolution feature map based on the upsampling kernel corresponding to each pixel in the target query feature map to obtain an upsampled feature map.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 11 are implemented.

Images