Image processing method and device and electronic equipment

Abstract

The invention provides an image processing method and device and electronic equipment, and the method comprises the steps: obtaining a first image which is an image formed by projecting an image on a spherical surface to a plane; performing convolution processing on the first image to obtain a first feature image; determining the position of each pixel in the first image projected from the spherical surface to the first image; and generating a second image based on the first feature image and the plurality of positions.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image processing technology, and in particular to an image processing method, device and electronic device. Background Art

[0002] Spherical super-resolution reconstruction refers to the process of restoring a spherical image from a certain resolution to a higher resolution. For example, a fisheye camera can capture a low-resolution spherical image, and electronic devices can improve the clarity of the spherical image based on spherical super-resolution reconstruction.

[0003] At present, electronic devices can learn the ability of image super-resolution based on convolutional neural networks (CNN), and perform super-resolution processing on 2D spherical images based on the trained convolutional neural network, thereby improving the clarity of spherical images. However, the uniformity of the distribution of texture information of spherical images is poor. For example, there is more texture information at the equator and less texture information at the poles. When spherical super-resolution reconstruction is performed based on CNN, the accuracy of spherical image super-resolution is poor, which leads to poor spherical image enhancement effect. Summary of the invention

[0004] The present disclosure provides an image processing method, device and electronic device, which are used to solve one or more technical problems in the prior art.

[0005] In a first aspect, the present disclosure provides an image processing method, the method comprising:

[0006] Acquire a first image, where the first image is an image projected from an image on a spherical surface onto a plane;

[0007] Performing convolution processing on the first image to obtain a first feature image;

[0008] Determine a position of each pixel in the first image projected from the spherical surface into the first image;

[0009] A second image is generated based on the first feature image and a plurality of locations.

[0010] In a second aspect, the present disclosure provides an image processing device, the image processing device comprising an acquisition module, a processing module, a determination module and a generation module, wherein:

[0011] The acquisition module is used to acquire a first image, where the first image is an image projected from an image on a spherical surface onto a plane;

[0012] The processing module is used to perform convolution processing on the first image to obtain a first feature image;

[0013] The determination module is used to determine the position of each pixel in the first image projected from the spherical surface to the first image;

[0014] The generating module is used to generate a second image based on the first feature image and a plurality of positions.

[0015] In a third aspect, an embodiment of the present disclosure provides an electronic device including: a processor and a memory;

[0016] The memory stores computer-executable instructions;

[0017] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the image processing method as described in the first aspect and various possible aspects of the first aspect.

[0018] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the image processing method as described in the first aspect and various possible aspects of the first aspect are implemented.

[0019] The present disclosure provides an image processing method, device and electronic device, the electronic device can obtain a first image, wherein the first image can be an image projected from an image on a spherical surface onto a plane, perform convolution processing on the first image to obtain a first feature image, determine the position of each pixel in the first image projected from the spherical surface into the first image, and generate a second image based on the first feature image and multiple positions. In the above method, since the electronic device can generate the second image by combining the position of the pixel in the first image projected from the spherical surface into the first image and the first feature image corresponding to the first image, the image quality of the second image reconstructed by the electronic device is greater than the image quality of the first image, thereby improving the image enhancement effect of the first image, improving the visualization effect of the second image, and improving the accuracy of the second image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0022] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0023] Figure 3 A schematic diagram of a first image provided by an embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of a process for obtaining a first feature image provided by an embodiment of the present disclosure;

[0025] Figure 5 A schematic diagram of a deformable convolution provided in an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of a method for generating a second image provided by an embodiment of the present disclosure;

[0027] Figure 7 A schematic diagram of a plurality of pixel groups provided in an embodiment of the present disclosure;

[0028] Figure 8 A schematic diagram of determining an offset associated with a pixel group provided in an embodiment of the present disclosure;

[0029] Fig. 9 A schematic diagram of updating a convolution kernel of a deformable convolution provided in an embodiment of the present disclosure;

[0030] Fig.10 A processing schematic diagram of a deformable convolution provided in an embodiment of the present disclosure;

[0031] Fig.11 A schematic diagram of an image processing method provided by an embodiment of the present disclosure;

[0032] Fig.12 A schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure; and

[0033] Fig.13 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0034] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0035] To facilitate understanding, the concepts involved in the embodiments of the present disclosure are explained below.

[0036] Electronic device: a device with wireless transceiver function. Electronic devices can be deployed on land, including indoors or outdoors, handheld, wearable or vehicle-mounted; they can also be deployed on the water (such as ships, etc.). The electronic device can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a virtual reality (VR) electronic device, an augmented reality (AR) electronic device, a wireless terminal in industrial control, a vehicle-mounted electronic device, a wireless terminal in self-driving, a wireless electronic device in remote medical, a wireless electronic device in smart grid, a wireless electronic device in transportation safety, a wireless electronic device in smart city, a wireless electronic device in smart home, a wearable electronic device, etc. The electronic device involved in the embodiments of the present disclosure can also be called a terminal, a user equipment (UE), an access electronic device, a vehicle-mounted terminal, an industrial control terminal, a UE unit, a UE station, a mobile station, a mobile station, a remote station, a remote electronic device, a mobile device, a UE electronic device, a wireless communication device, a UE agent or a UE device, etc. Electronic devices may also be fixed or mobile.

[0037] Next, combine Figure 1 , the application scenarios of the embodiments of the present disclosure are described.

[0038] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present disclosure. Figure 1 , including: an electronic device and a virtual reality device. Among them, the display page of the electronic device may include a low-definition spherical image, and the electronic device may perform spherical super-resolution reconstruction processing on the low-definition spherical image to obtain a high-definition spherical image. The electronic device may send a high-definition spherical image to the virtual reality device, and the virtual reality device may play the high-definition spherical image. In this way, the electronic device can improve the visualization effect of the spherical image in the virtual reality device based on the method of spherical super-resolution reconstruction, and improve the user experience.

[0039] It should be noted that the spherical image in the embodiment of the present disclosure may be a plane image. For example, the spherical image in the embodiment of the present disclosure may be an equirectangular projection (ERP) image after the electronic device performs ERP processing on the three-dimensional spherical image, and the embodiment of the present disclosure is not limited to this.

[0040] It should be noted that the low-definition spherical image in the embodiment of the present disclosure may include a spherical image with a higher degree of blur, a spherical image with a lower resolution, a spherical image with poor visualization, and a spherical image with poor clarity, etc., and the embodiment of the present disclosure is not limited to this. The high-definition spherical image in the embodiment of the present disclosure may include a spherical image with a higher degree of sharpness, a spherical image with a higher resolution, a spherical image with better visualization, and a spherical image with higher clarity, etc., and the embodiment of the present disclosure is not limited to this.

[0041] It should be noted that Figure 1 The application scenarios of the embodiments of the present disclosure are merely illustrative and are not intended to limit the application scenarios of the embodiments of the present disclosure.

[0042] In the related art, a virtual reality device (VR) can present a three-dimensional display effect based on a spherical image (i.e., displaying an image on a spherical surface, with the user's perspective located at the center of the sphere). In order to improve the display effect of a spherical image in a virtual display device, the electronic device can reconstruct a low-definition spherical image into a high-definition spherical image based on the technology of spherical super-resolution reconstruction, and send the high-definition spherical image to the VR device, so that the three-dimensional scene displayed by the VR device has a higher definition. At present, electronic devices can process a planar spherical image based on a trained super-resolution model to obtain a spherical image with higher definition. For example, a CNN module in a super-resolution model can learn the ability of image super-resolution, so a spherical image with higher definition can be obtained by convolution processing the spherical image based on the CNN module. However, due to the poor uniformity of the texture information distribution of the spherical image, for example, the texture information at the equatorial position of the spherical image is more, and the texture information at the polar positions is less, therefore, the accuracy of the CNN module convolution is poor, resulting in poor accuracy of the spherical image super-resolution, which in turn leads to poor effect of spherical image enhancement.

[0043] In order to solve the technical problems in the related art, the embodiment of the present disclosure provides an image processing method, in which an electronic device can obtain a first image projected from an image on a spherical surface onto a plane, and perform convolution processing on the first image to obtain a first feature image, determine the position of each pixel in the first image projected from the spherical surface into the first image, encode the position of each pixel in the first image to obtain a position code, perform convolution processing on multiple position codes of each group of pixels in the first image, and obtain a bias feature corresponding to each group of pixels, wherein the bias feature includes the offset of each convolution kernel in the deformable convolution, and generate a second image based on the deformable convolution, multiple bias features and the first feature image. In the above method, since the electronic device can generate the second image by combining multiple position codes of pixels in the first image and the first feature image associated with the first image, the image quality of the second image is high, and the electronic device can flexibly adjust the convolution position of the convolution kernel of the deformable convolution based on the position code, so the deformable convolution can accurately perform convolution processing on the first feature image, thereby improving the accuracy of the convolution, improving the accuracy of the generated second image, and improving the image enhancement effect.

[0044] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0045] Figure 2 A flowchart of an image processing method provided by an embodiment of the present disclosure. Figure 2 , the method may include:

[0046] S201: Acquire a first image.

[0047] The execution subject of the embodiments of the present disclosure may be an electronic device, or an image processing device provided in the electronic device. Among them, the image processing device may be implemented based on software, or the image processing device may be implemented based on a combination of software and hardware, which is not limited in the embodiments of the present disclosure. Optionally, the electronic device may be any device with end-to-end computing capabilities. For example, the electronic device may be a computer, a server, or other device, which is not limited in the embodiments of the present disclosure.

[0048] The first image may be an image to be enhanced. Image enhancement may improve the resolution, clarity, and visualization of an image. For example, in an image enhancement scenario of a first image, the electronic device may enhance the visualization of the first image based on image super-resolution reconstruction. For example, the resolution of the first image may be 200*400, and after the electronic device performs super-resolution reconstruction on the first image, the resolution of the obtained image may be 800*1600, which may improve the visualization of the image.

[0049] Optionally, the first image may be an image projected from an image on a spherical surface onto a plane. For example, the first image may be a two-dimensional plane image. For example, in a VR scene, a spherical image may be set on the surface of a spherical model, and the user's perspective is the center of the sphere, so that a three-dimensional display effect can be presented to the user. When performing image enhancement processing on the spherical image, the electronic device may convert the three-dimensional image set on the surface of the spherical model into a two-dimensional image on a plane by means of ERP projection, and the electronic device may determine the two-dimensional image on the plane as the first image.

[0050] Next, combine Figure 3 , the spherical image in the embodiment of the present disclosure is explained.

[0051] Figure 3 A schematic diagram of a first image provided by an embodiment of the present disclosure. Figure 3 , including: a three-dimensional sphere and a two-dimensional plane. The surface of the three-dimensional sphere includes a three-dimensional image, and the user's perspective is located at the center of the sphere to present a three-dimensional display effect of the image. The image on the surface of the three-dimensional sphere can be converted to a two-dimensional plane by projection (such as an ERP projector), that is, the image on the surface of the three-dimensional sphere is unfolded on the two-dimensional plane to obtain a two-dimensional spherical image. The first image in the embodiment of the present disclosure can be a spherical image on the two-dimensional plane.

[0052] Optionally, the electronic device may obtain the first image from a database. For example, the database may pre-store multiple images to be enhanced, and when the electronic device receives an image enhancement task, it may obtain multiple first surface images to be processed from the database.

[0053] Optionally, the electronic device may receive the first image sent by another device. For example, since the image captured by the fisheye camera may be a planar image after ERP projection, the electronic device may receive the first image sent by the fisheye camera. For example, after the fisheye camera captures multiple first images, the multiple first images may be stored in a database, and the electronic device may obtain the first image in the database.

[0054] It should be noted that the electronic device can acquire the first image based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0055] S202: Perform convolution processing on the first image to obtain a first feature image.

[0056] Among them, the first feature image may be a feature image associated with the image information of the first image. For example, the first feature image may include pixel information, color features, texture features, and spatial relationship features of the first image, which is not limited in the embodiments of the present disclosure. For example, the first feature image may be related to the pixel distribution information of the first image, and the texture information, pixel information, etc. of the first image may be determined based on the first feature image. Optionally, the first feature image may be represented by a matrix. For example, if the width of the first image is W, the height is H, and the number of convolution channels is N, then the first feature image may be a matrix of [N, H, W].

[0057] Optionally, the electronic device may perform convolution processing on the first image based on a convolutional neural network CNN, thereby obtaining a first feature image associated with the first image. For example, during the training of the convolutional neural network CNN, the CNN may learn the ability to extract image information of the first image, so after the training of the convolutional neural network is completed, the electronic device may input the first image into the convolutional neural network, and the convolutional neural network may output a first image feature associated with the image information of the first image.

[0058] Next, combine Figure 4 , the process of obtaining the first feature image is explained.

[0059] Figure 4 A schematic diagram of a process for obtaining a first feature image provided by an embodiment of the present disclosure. Figure 4 , including: a first image and a feature extraction module. The first image may be a two-dimensional plane image, the feature extraction module may be a convolutional neural network CNN, and the feature extraction module may be a convolution block including a 1*1 convolution and an arbitrary activation function combination. The feature extraction module learns the ability to extract feature maps associated with image information during training. Electronic device ( Figure 4 The first image may be input to the feature extraction module (not shown), and the feature extraction module may process the first image to obtain a first feature image associated with the first image, and the first feature image may be represented based on a matrix. For example, the shape of the first feature image may be [N, H, W], where N is the number of channels of the convolutional neural network, H is the height of the first image, and W is the width of the first image.

[0060] S203 , determining a position of each pixel in the first image projected from the spherical surface to the first image.

[0061] Among them, the position of the pixel can be the position of the pixel projected from the spherical surface to the first image. For example, since the first image in the embodiment of the present disclosure is an image of a spherical image projected onto a two-dimensional plane, the position of each pixel can also indicate the position of the pixel in the sphere after the spherical image is reconverted into a three-dimensional spherical image. For example, in a VR scene, the two-dimensional first image is converted into a three-dimensional spherical image when displayed, and the position in the spherical image can be determined based on the position of the pixel in the two-dimensional first image. For example, based on the ERP projection method, the spherical image is processed to obtain the first image, wherein the electronic device determines the position of the pixel in the sphere based on the position of the pixel in the first image, that is, the position of the pixel in the plane first image corresponds to the position of the pixel in the spherical image corresponding to the first image.

[0062] Optionally, the electronic device may determine the position of each pixel in the first image projected from the spherical surface to the first image based on the projection method. For example, the electronic device projects the spherical image into the first image based on the ERP projection method. If pixel A is a pixel with an angle of 1 in the spherical image, then the pixel A is projected into the first image. The electronic device may determine that pixel A may be the first pixel in the first image based on the rule of ERP projection. For example, after the electronic device acquires the first image, it may obtain the ERP projection information associated with the first image. The ERP projection information may include the horizontal coordinate of the pixel in the first image, the vertical coordinate of the pixel in the first image, the distance of the pixel from the origin in the sphere, the angle between the pixel in the sphere and the positive direction of the horizontal axis, etc. The embodiments of the present disclosure are not limited to this.

[0063] It should be noted that the electronic device may also determine the position of each pixel in the first image projected from the spherical surface to the first image based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0064] S204: Generate a second image based on the first feature image and the multiple positions.

[0065] The second image may be an image obtained by performing image enhancement processing on the first image. Optionally, image enhancement may include image denoising, image super-resolution reconstruction, image restoration, etc.

[0066] Image denoising can be filtering the noise in the image to improve the image display effect. For example, a low-quality image includes some noise, which can interfere with the details and features of the image. Therefore, denoising the image can enhance the image display effect. The image denoising method can include mean filtering, median filtering, small filtering, etc.

[0067] Among them, image super-resolution reconstruction can be a process of restoring a low-resolution image to a high-resolution image. For example, in a low-quality image, due to the small number of pixels, image details and features are difficult to express, so the number of pixels of the image can be increased by an image super-resolution reconstruction method, wherein the image super-resolution reconstruction method may include an interpolation method, a method based on deep learning, etc.

[0068] Image restoration is the process of processing a damaged image to restore the original information of the image. For example, in a low-quality image, due to the influence of various factors, the image has problems such as distortion and blur. Therefore, the image can be processed by image restoration methods such as inverse filtering and least squares method to improve the display effect of the image.

[0069] The image quality of the second image is greater than that of the first image. For example, the clarity of the second image is higher than that of the first image, the resolution of the second image is higher than that of the first image, and the visualization effect of the second image is higher than that of the first image. The embodiments of the present disclosure are not limited to this.

[0070] The electronic device may generate the second image based on the following feasible implementation method: encoding the position of each pixel in the first image to obtain multiple position codes, and generating the second image based on the multiple position codes and the first feature image.

[0071] Optionally, the pixel position encoding may include an absolute position encoding of the pixel and a relative position encoding of the pixel. The absolute position encoding may be an encoding associated with the absolute position of the pixel, and the relative position encoding may be an encoding associated with the relative position of the pixel. For example, the electronic device may obtain the absolute position of each pixel, and then encode the absolute position coordinates by a preset encoding method, and then obtain the absolute position encoding. For example, the electronic device may process the position of the pixel based on sinusoidal position encoding to obtain a relative position encoding.

[0072] Optionally, the electronic device may obtain multiple position codes based on the following feasible implementation methods: encoding the position of each pixel to obtain an absolute position code for each pixel, encoding the position of each pixel based on sine and cosine functions to obtain a relative position code for each pixel.

[0073] Optionally, the electronic device may determine the absolute position code of each pixel based on the following formula:

[0074]

[0075]

[0076] Among them, position_mat is the absolute position code of the pixel; width is the width of the first image; height is the height of the first image; position_w is the horizontal coordinate of the pixel in the first image; position_h is the vertical coordinate of the pixel in the first image; d is the dimension of the encoding vector.

[0077] Optionally, the electronic device may determine the relative position code of the pixel based on the following formula:

[0078]

[0079]

[0080] position=PE(i, 2j)×PE(i, 2j+1)

[0081] Among them, d is the latitude of the encoding vector; i is the horizontal coordinate or vertical coordinate of the pixel; j is an integer from 0 to d / 2; position is the relative position code associated with the pixel.

[0082] The dimension of the coding vector may indicate the number of coding vectors associated with the position of the pixel. For example, if the dimension of the coding vector is 1000, the number of relative position codes of the pixel is 1000, and if the dimension of the coding vector is 2000, the number of relative position codes of the pixel is 2000. It should be noted that the larger the dimension of the coding vector, the higher the accuracy of the relative position code of the pixel, but the lower the efficiency of determining the relative position code (the amount of calculation increases).

[0083] For example, a pixel sequence with a length of L (for example, the width of the first image is 200, then L is 200), d is 1000, and for the 10th pixel, j is sequentially varied from 0 to 500. Each time j takes a value, the sin value and cos value corresponding to the 10th pixel can be obtained. By multiplying the sin value and the cos value, a relative position code corresponding to the 10th pixel can be obtained. After j is varied from 0 to 500, all relative position codes corresponding to the 10th pixel can be obtained.

[0084] For example, in actual application, for any pixel, the horizontal coordinate coding vector corresponding to the horizontal coordinate of the pixel and the vertical coordinate coding vector corresponding to the vertical coordinate of the pixel can be obtained based on the above formula, and the electronic device can splice all the horizontal coordinate coding vectors and vertical coordinate coding vectors of the pixel to obtain the relative position coding of the pixel. In this way, the electronic device can not only obtain the absolute position coding of the pixel, but also implicitly express the relative position coding of the pixel, thereby improving the accuracy of the pixel position coding.

[0085] It should be noted that the electronic device can determine the relative position coding and absolute position coding of the pixel based on any feasible implementation method (such as encoding the position of the pixel based on the BERT model), and the embodiments of the present disclosure are not limited to this.

[0086] Optionally, the electronic device generates a second image based on multiple position codes and the first feature image. Specifically, it can be: convolution processing is performed on multiple position codes of each group of pixels in the first image to obtain bias features corresponding to each group of pixels, and the second image is generated based on deformable convolution, multiple bias features and the first feature image.

[0087] The bias feature may include an offset of each convolution kernel in the deformable convolution. For example, the electronic device may perform convolution processing on multiple position codes of each group of pixels based on a convolutional neural network, and then obtain a bias feature corresponding to each group of pixels.

[0088] Optionally, the first image may include multiple pixel groups, and therefore, the first feature image may also include multiple pixel groups, and each pixel group may be associated with a bias feature. For example, the first image includes 100 pixels, each pixel group may include 10 pixels, and the first feature image corresponding to the first image may also include 100 pixels, each pixel group may also include 10 pixels, wherein some pixels in multiple pixel groups may overlap, and the embodiments of the present disclosure are not limited to this.

[0089] Among them, deformable convolution is a convolution operation in a convolutional neural network. Deformable convolution can perform nonlinear deformation on the input feature map during the convolution process to adapt to the shape of the object in the image. Deformable convolution has a better convolution effect on images of irregularly shaped objects (such as images of animals, images of vehicles, etc.).

[0090] Among them, in deformable convolution, the shape of the convolution kernel is variable (the shape of the convolution kernel is fixed during the traditional convolution operation), and the way of changing the shape of the convolution kernel can be learned during the training process. For example, deformable convolution can introduce a set of offsets to control the shapes of multiple convolution kernels. The offsets can be learned based on network learning. When performing convolution operations, deformable convolution can perform nonlinear deformation on the input feature map based on the offsets to adapt to the shape of the object in the image.

[0091] The offset may be an offset of a convolution kernel in a deformable convolution. For example, for any convolution kernel in a deformable convolution, the offset of the convolution kernel may include an offset of the convolution kernel in an x-direction and an offset of the convolution kernel in a y-direction.

[0092] Optionally, the electronic device may determine a set of offsets corresponding to the pixel group based on multiple position codes corresponding to the pixel group. For example, if the pixel group includes 10 pixels, the electronic device may obtain the position codes of the 10 pixels and perform convolution processing on the 10 position codes to obtain a set of offsets corresponding to the 10 pixels, wherein the set of offsets may include an x ​​offset and a y offset of each convolution kernel of the deformable convolution.

[0093] It should be noted that the electronic device can determine the pixel group based on the size of each convolution of the deformable convolution in the first feature image. For example, if the deformable convolution convolves 100 pixels each time, each pixel group may include 100 pixels. The electronic device can also determine multiple pixel groups in the first feature image based on any feasible implementation method, and the embodiments of the present disclosure are not limited to this.

[0094] Next, combine Figure 5 , explaining deformable convolution.

[0095] Figure 5 A schematic diagram of a deformable convolution provided in an embodiment of the present disclosure. Figure 5 , including a convolution block. Among them, the convolution block can include 9 convolution kernels. Before deformation based on the offset, the convolution block is arranged in the same way as the traditional convolution, and both perform convolution processing on the image based on the 3*3 convolution kernel. The convolution block can be deformed based on the offset, and the 9 convolution kernels in the convolution block can be moved to different positions, thereby achieving the deformable effect of the convolution kernel. After the convolution kernel is deformed, it can match the texture information of the image, so that the convolution effect can be improved. For example, if the image includes irregular objects, the traditional convolution block can only perform convolution processing according to the convolution kernel in a fixed position, while the deformable convolution can match the position of the convolution kernel with the irregular object during each convolution, which can improve the efficiency and accuracy of the convolution.

[0096] Optionally, the electronic device can perform convolution processing on the first feature image based on the deformable convolution and multiple bias features, and after the convolution is completed, perform feature reconstruction on the convolution result to obtain the second image. In this way, since the deformable convolution has high convolution accuracy and high convolution efficiency, the electronic device can obtain an accurate second image, improve the acquisition efficiency of the second image and the accuracy of the second image.

[0097] The disclosed embodiment provides an image processing method, wherein an electronic device can obtain a first image projected from an image on a spherical surface onto a plane, and perform convolution processing on the first image to obtain a first feature image, determine the position of each pixel in the first image projected from the spherical surface into the first image, encode the position of each pixel in the first image to obtain a position code, perform convolution processing on multiple position codes of each group of pixels in the first image to obtain a bias feature corresponding to each group of pixels, wherein the bias feature includes an offset of each convolution kernel in a deformable convolution, and generate a second image based on the deformable convolution, multiple bias features and the first feature image. In the above method, since the electronic device can generate the second image by combining multiple position codes of pixels in the first image and the first feature image associated with the first image, the image quality of the second image is high, and the electronic device can flexibly adjust the convolution position of the convolution kernel of the deformable convolution based on the position code, so the deformable convolution can accurately perform convolution processing on the first feature image, thereby improving the accuracy of the convolution, improving the accuracy of the generated second image, and improving the image enhancement effect.

[0098] exist Figure 2 Based on the embodiment shown below, combined with Figure 6 , a method for generating a second image based on deformable convolution, multiple bias features and a first feature image in the above-mentioned image processing method is described.

[0099] Figure 6 A schematic diagram of a method for generating a second image provided by an embodiment of the present disclosure. Figure 6 The method flow may include:

[0100] S601: Perform convolution processing on a first feature image based on deformable convolution and multiple bias features to obtain a second feature image.

[0101] Optionally, the first feature image may include multiple pixel groups, and the deformable convolution may perform convolution processing on multiple pixels in a pixel group each time.

[0102] Next, combine Figure 7 , describe multiple pixel groups in the first feature image.

[0103] Figure 7 A schematic diagram of a plurality of pixel groups provided by an embodiment of the present disclosure. Figure 7, including: a first feature image. The first feature image may include region 1, region 2, region 3, region 4, and region 5. Each region may be a region where deformable convolution performs convolution processing each time. Multiple pixels in region 1 may be a pixel group, multiple pixels in region 2 may be a pixel group, multiple pixels in region 3 may be a pixel group, multiple pixels in region 4 may be a pixel group, and multiple pixels in region 5 may be a pixel group.

[0104] It should be noted that Figure 7 It is only an exemplary illustration of a partial pixel group in the first feature image, not a limitation on the pixel group. The deformable convolution operation can traverse all pixels in the first feature image. Therefore, multiple pixel groups in the first feature image can cover all pixels, and the shapes related to the multiple pixel groups in the first feature image can be any shape, which is not limited in this embodiment of the present disclosure.

[0105] Optionally, each pixel group can be associated with a set of offsets. Figure 7 In the illustrated embodiment, a plurality of pixels in region 1 may be associated with a set of offsets, and a plurality of pixels in region 2 may be associated with another set of offsets. The number of each set of offsets is associated with the convolution kernel of the deformable convolution. For example, if the deformable convolution includes 9 convolution kernels, each set of offsets may include 9 horizontal coordinate offsets and 9 vertical coordinate offsets. If the deformable convolution includes 25 convolution kernels, each set of offsets may include 25 horizontal coordinate offsets and 25 vertical coordinate offsets.

[0106] Among them, the electronic device can perform convolution processing on the first feature image based on the following feasible implementation method: for any bias feature, based on the bias feature, the position of each convolution kernel in the deformable convolution is updated to obtain an updated deformable convolution, and based on the updated deformable convolution, the first feature image is convolution processed.

[0107] Optionally, the electronic device may determine a set of offsets corresponding to the bias features based on the bias features, and then update the position of each convolution kernel in the deformable convolution based on the offsets to obtain an updated deformable convolution. The offsets may include a horizontal axis offset and a vertical axis offset. For example, Figure 7 In the illustrated embodiment, region 1 may include a plurality of pixels, and the electronic device may obtain position codes of the plurality of pixels in region 1, and then determine bias features corresponding to the plurality of pixels in region 1 based on the position codes of the plurality of pixels. The electronic device may obtain a set of x-direction offsets and a set of y-direction offsets associated with the plurality of pixels in region 1 based on the bias features, that is, the offset of each convolution kernel in the deformable convolution.

[0108] Optionally, the convolutional neural network that extracts bias features can learn the ability to extract offset features in position codes during training. Therefore, the electronic device can process the position codes based on the convolutional neural network to obtain a set of offsets. For example, if the deformable convolution includes 9 convolution kernels, the electronic device can input multiple position codes of multiple pixels in the pixel group to the CNN, and the convolutional neural network can output 9 x-direction offsets and 9 y-direction offsets, wherein an x-direction offset and the corresponding y-direction offset can be the offset of a convolution kernel in the deformable convolution. In this way, the electronic device can accurately determine the offset corresponding to the deformable convolution based on multiple position codes of the pixel group, thereby improving the efficiency of the deformable convolution on the pixel group and the accuracy of the convolution.

[0109] Next, combine Figure 8 , describes a process by which an electronic device determines a set of offsets associated with pixel values.

[0110] Figure 8 A schematic diagram of determining an offset associated with a pixel group provided by an embodiment of the present disclosure. Figure 8 , including pixel 1, pixel 2, ..., pixel n. Among them, n pixels can be a group of pixels that are convolved by deformable convolution. Electronic device ( Figure 8 The electronic device may input n position codes into the convolution block, and the convolution block may extract offset features from the n position codes to obtain a set of offsets.

[0111] Optionally, the electronic device performs convolution processing on the first feature image based on the updated deformable convolution, which can be specifically: based on the updated deformable convolution, convolution processing is performed on the pixels to be convolved in the first feature image, wherein the pixels to be convolved can be the pixels corresponding to a group of pixels corresponding to the bias feature in the first feature image. For example, if the electronic device determines a bias feature based on the 1st pixel, the 2nd pixel, ..., the 9th pixel in the first image, and obtains a set of offsets based on the bias feature, after the electronic device performs convolution kernel offset on the deformable convolution based on the offset, the pixels to be convolved in the first feature image of the deformable convolution can be the 9 pixels corresponding to the 1st pixel, the 2nd pixel, ..., the 9th pixel in the first image in the first feature image.

[0112] Among them, the electronic device can determine the offset corresponding to each convolution kernel in the deformable convolution, and then transform the position of the convolution kernel based on the offset. For example, if the deformable convolution may include convolution kernel 1, convolution kernel 2, convolution kernel 3 and convolution kernel 4, then the set of offsets obtained by the electronic device may include the offsets in the x-direction and y-direction corresponding to convolution kernel 1, the offsets in the x-direction and y-direction corresponding to convolution kernel 2, the offsets in the x-direction and y-direction corresponding to convolution kernel 3, and the offsets in the x-direction and y-direction corresponding to convolution kernel 4.

[0113] Optionally, the electronic device may update the position of the convolution kernel in the deformable convolution. For example, if the electronic device determines that the offset corresponding to the convolution kernel is x moved 2 pixel units to the left and y moved 5 pixel units downward, the electronic device may move convolution kernel 1 2 pixel units to the left and 5 pixel units downward, thereby completing the update of the convolution kernel.

[0114] Next, combine Fig. 9 , the updating process of the convolution kernel of deformable convolution is explained.

[0115] Fig. 9 A schematic diagram of a convolution kernel update for a deformable convolution provided by an embodiment of the present disclosure. Fig. 9 , including: a convolution block. The convolution block may include a convolution kernel A, a convolution kernel B, a convolution kernel C and a convolution kernel D. Electronic device ( Fig. 9 The offsets obtained by the convolution kernel (not shown) may include offset 1 corresponding to convolution kernel A: x is translated 2 units to the right, offset 2 corresponding to convolution kernel B: x is translated 2 units to the left, offset 3 corresponding to convolution kernel C: x is translated 2 units to the right, y is translated 1 unit upward, and offset 4 corresponding to convolution kernel D: x is translated 2 units to the left. The electronic device may update the positions of convolution kernel A, convolution kernel B, convolution kernel C, and convolution kernel D based on the offsets to obtain an updated convolution block.

[0116] Next, combine Fig.10 , the process of deformable convolution performing convolution processing on the convolved pixels is explained.

[0117] Fig.10 A processing diagram of a deformable convolution provided by an embodiment of the present disclosure. Fig.10 , including: a first feature image and a convolution block. The first image feature includes a region to be convolved, and the convolution block may include 9 convolution kernels. Electronic device ( Fig.10(not shown) can determine a set of offsets associated with the plurality of pixels in the area to be convolved based on the position encoding of the plurality of pixels in the area to be convolved in the first feature image, and update the positions of the nine convolution kernels in the convolution block based on the set of offsets. The electronic device can perform convolution processing on the area to be convolved based on the convolution block after the position update.

[0118] Optionally, the electronic device may obtain a second feature image after convolution processing on all pixels in the first feature image based on multiple updated deformable convolutions corresponding to multiple bias features. For example, the first image may include pixel group A and pixel group B. The electronic device obtains a set of offsets 1 based on multiple position codes of pixel group A, and obtains another set of offsets 2 based on multiple position codes of pixel group B. The electronic device updates the position of the convolution kernel of the deformable convolution based on a set of offsets 1 to obtain an updated deformable convolution a. The electronic device updates the position of the convolution kernel of the deformable convolution based on another set of offsets 2 to obtain an updated deformable convolution b. The electronic device performs convolution processing on multiple pixels corresponding to pixel group A in the first feature image based on deformable convolution a, and performs convolution processing on multiple pixels corresponding to pixel group B in the first feature image based on deformable convolution b, thereby obtaining a second feature image.

[0119] S602: Reconstruct the second feature image to obtain a second image.

[0120] After the electronic device obtains the second feature image, the electronic device can perform feature reconstruction processing based on the convolutional neural network to obtain the second image. For example, the convolutional neural network can learn the ability to reconstruct images based on feature images during training. Therefore, the electronic device can input the second feature image to the convolutional neural network, and the convolutional neural network can perform image reconstruction processing based on the second feature image to obtain the second image.

[0121] The disclosed embodiment provides a method for generating a second image, wherein the electronic device can perform convolution processing on a first feature image based on a deformable convolution and a plurality of bias features to obtain a second feature image, and reconstruct the second feature image to obtain a second image. In this way, since the position of the convolution kernel of the deformable convolution can be changed, the deformable convolution has a high degree of adaptability to the first feature image, thereby improving the accuracy and efficiency of the convolution of the first feature image, improving the accuracy of the second feature image, and thereby improving the accuracy of the second image reconstructed based on the second feature image.

[0122] Based on any of the above embodiments, Fig.11 , the process of the above-mentioned image processing method is explained.

[0123] Fig.11The following is a schematic diagram of an image processing method provided by an embodiment of the present disclosure. Fig.11 , including: a first image, a feature extraction module, a deformable convolution, a bias feature extraction module and a feature reconstruction module. The feature extraction module can obtain a feature image associated with image information, the bias feature extraction module can obtain a bias feature including an offset based on multiple position encodings, and the feature reconstruction module can reconstruct a new image based on the feature image.

[0124] See also Fig.11 , electronic equipment ( Fig.11 (not shown) can process the first image based on the feature extraction module to obtain a first feature image associated with the first image, and the electronic device can determine the position code of each pixel, wherein the position code can include a relative position code and an absolute position code.

[0125] See also Fig.11 The electronic device may input the position code to the bias feature extraction module, and the bias feature extraction module may output multiple groups of offsets. The electronic device may update the position of the convolution kernel of the deformable convolution based on each group of offsets, and perform convolution processing on multiple pixels associated with the offsets in the first feature image based on the updated deformable convolution.

[0126] See also Fig.11 After the electronic device performs convolution processing on all pixels in the first feature image based on the deformable convolution, a second feature image can be obtained. The electronic device can perform image reconstruction processing on the second feature image based on the feature reconstruction module to obtain a second image. In this way, when the electronic device reconstructs the second image, the image features of the first image and the position of each pixel in the first image can be combined, thereby improving the accuracy of the second image.

[0127] Fig.12 This is a schematic diagram of the structure of an image processing device provided by an embodiment of the present disclosure. Fig.12 The image processing device 120 includes an acquisition module 121, a processing module 122, a determination module 123 and a generation module 124, wherein:

[0128] The acquisition module 121 is used to acquire a first image, where the first image is an image projected from an image on a spherical surface onto a plane;

[0129] The processing module 122 is used to perform convolution processing on the first image to obtain a first feature image;

[0130] The determination module 123 is used to determine the position of each pixel in the first image projected from the spherical surface to the first image;

[0131] The generating module 124 is configured to generate a second image based on the first feature image and a plurality of positions.

[0132] According to one or more embodiments of the present disclosure, the generating module 124 is specifically used for:

[0133] Encoding the position of each pixel in the first image to obtain a plurality of position codes;

[0134] The second image is generated based on the multiple position codes and the first feature image.

[0135] According to one or more embodiments of the present disclosure, the generating module 124 is specifically used for:

[0136] Performing convolution processing on multiple position codes of each group of pixels in the first image to obtain bias features corresponding to each group of pixels, wherein the bias features include an offset of each convolution kernel in the deformable convolution;

[0137] The second image is generated based on the deformable convolution, a plurality of bias features, and the first feature image.

[0138] According to one or more embodiments of the present disclosure, the generating module 124 is specifically used for:

[0139] Based on the deformable convolution and the plurality of bias features, convolution processing is performed on the first feature image to obtain a second feature image;

[0140] The second feature image is reconstructed to obtain the second image.

[0141] According to one or more embodiments of the present disclosure, the generating module 124 is specifically used for:

[0142] For any bias feature, based on the bias feature, the position of each convolution kernel in the deformable convolution is updated to obtain an updated deformable convolution;

[0143] Based on the updated deformable convolution, convolution processing is performed on the first feature image.

[0144] According to one or more embodiments of the present disclosure, the generating module 124 is specifically used for:

[0145] Based on the updated deformable convolution, convolution processing is performed on pixels to be convolved in the first feature image, where the pixels to be convolved are pixels corresponding to a group of pixels corresponding to the bias feature in the first feature image.

[0146] According to one or more embodiments of the present disclosure, the generating module 124 is specifically used for:

[0147] Encoding the position of each pixel to obtain an absolute position code of each pixel;

[0148] The position of each pixel is encoded based on sine and cosine functions to obtain a relative position code of each pixel; the pixel position code includes the absolute position code and the relative position code of the pixel.

[0149] The image processing device provided in the embodiment of the present disclosure may be used to execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0150] Fig.13 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Fig.13 , which shows a schematic diagram of the structure of an electronic device 1300 suitable for implementing the embodiment of the present disclosure. The electronic device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.13 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0151] like Fig.13 As shown, the electronic device 1300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage device 1308 to a random access memory (RAM) 1303. Various programs and data required for the operation of the electronic device 1300 are also stored in the RAM 1303. The processing device 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0152] Typically, the following devices may be connected to the I / O interface 1305: an input device 1306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 1307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 1308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1309. The communication device 1309 may allow the electronic device 1300 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.13 The electronic device 1300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0153] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 1309, or installed from a storage device 1308, or installed from a ROM 1302. When the computer program is executed by the processing device 1301, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0154] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0155] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0156] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0157] An embodiment of the present disclosure provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the image processing method that may be involved in various embodiments as described above is implemented.

[0158] An embodiment of the present disclosure provides a computer program product, including a computer program, and when the computer program is executed by a processor, the image processing method that may be involved in various embodiments as described above is implemented.

[0159] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0160] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0161] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0162] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0163] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0164] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0165] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0166] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0167] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can independently choose whether to provide personal information to software or hardware such as an electronic device, application, server or storage medium that performs the operation of the technical solution of the present disclosure based on the prompt message. As an optional but non-limiting implementation method, in response to receiving an active request from a user, the method of sending a prompt message to the user can be, for example, a pop-up window, and the prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0168] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0169] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of the relevant laws and regulations. The data may include information, parameters and messages, such as flow switching indication information.

[0170] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0171] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0172] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. An image processing method, characterized in that: include: Acquire a first image, where the first image is an image projected from an image on a spherical surface onto a plane; Performing convolution processing on the first image to obtain a first feature image; Determine a position of each pixel in the first image projected from the spherical surface into the first image; A second image is generated based on the first feature image and a plurality of locations.

2. The method according to claim 1, characterized in that The step of generating a second image based on the first feature image and a plurality of positions comprises: Encoding the position of each pixel in the first image to obtain a plurality of position codes; The second image is generated based on the multiple position codes and the first feature image.

3. The method according to claim 2, characterized in that The step of generating the second image based on the plurality of position codes and the first feature image comprises: Performing convolution processing on multiple position codes of each group of pixels in the first image to obtain bias features corresponding to each group of pixels, wherein the bias features include an offset of each convolution kernel in the deformable convolution; The second image is generated based on the deformable convolution, a plurality of bias features, and the first feature image.

4. The method according to claim 3, characterized in that The step of generating the second image based on the deformable convolution, a plurality of bias features and the first feature image comprises: Based on the deformable convolution and the plurality of bias features, convolution processing is performed on the first feature image to obtain a second feature image; The second feature image is reconstructed to obtain the second image.

5. The method according to claim 4, characterized in that The performing convolution processing on the first feature image based on the deformable convolution and the multiple bias features includes: For any bias feature, based on the bias feature, the position of each convolution kernel in the deformable convolution is updated to obtain an updated deformable convolution; Based on the updated deformable convolution, convolution processing is performed on the first feature image.

6. The method according to claim 5, characterized in that The step of performing convolution processing on the first feature image based on the updated deformable convolution includes: Based on the updated deformable convolution, convolution processing is performed on pixels to be convolved in the first feature image, where the pixels to be convolved are pixels corresponding to a group of pixels corresponding to the bias feature in the first feature image.

7. The method according to any one of claims 2 to 6, characterized in that: The step of encoding the position of each pixel in the first image to obtain a plurality of position codes includes: Encoding the position of each pixel to obtain an absolute position code of each pixel; The position of each pixel is encoded based on sine and cosine functions to obtain a relative position code of each pixel; the pixel position code includes the absolute position code and the relative position code of the pixel.

8. An image processing device, characterized in that: It includes an acquisition module, a processing module, a determination module and a generation module, wherein: The acquisition module is used to acquire a first image, where the first image is an image projected from an image on a spherical surface onto a plane; The processing module is used to perform convolution processing on the first image to obtain a first feature image; The determination module is used to determine the position of each pixel in the first image projected from the spherical surface to the first image; The generating module is used to generate a second image based on the first feature image and a plurality of positions.

9. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the image processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the image processing method according to any one of claims 1 to 7 is implemented.

Images