Panoramic Image Processing Method, Computer Device, and Storage Medium

Abstract

The present application relates to a panoramic image processing method, a computer device, a storage medium, and a computer program product. The method includes: performing deformable convolution processing on a panoramic image to be processed to obtain a feature image of the panoramic image; performing regression processing on the feature image to obtain image transformation information of the panoramic image; obtaining a mapped pixel position corresponding to the corrected pixel position in the panoramic image according to the image transformation information and the corrected pixel position of the panoramic image, where the corrected pixel position is represented as the coordinate position of a pixel in a corrected image corresponding to the panoramic image; and performing pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain a corrected image corresponding to the panoramic image. Using this method can improve the processing effect of the network on panoramic images.

Description

Technical Field

[0001] This application relates to the field of image processing technologies, and in particular, to a panoramic image processing method, a computer device, a storage medium, and a computer program product. Background Art

[0002] A panoramic camera can capture a wider field of view than a traditional lens. The panoramic camera can capture multi-view images such as the sides, top, and back that cannot be captured by a traditional lens.

[0003] However, the wide field of view also causes some distortions and aberrations in the imaging of panoramic images. The distorted imaging will result in poor effects when ordinary networks process panoramic images or panoramic videos. Summary of the Invention

[0004] Based on this, in view of the above technical problems, it is necessary to provide a panoramic image processing method, a computer device, a computer-readable storage medium, and a computer program product that can improve the processing effect of a network on panoramic images.

[0005] In a first aspect, this application provides a panoramic image processing method. The method includes:

[0006] Performing deformable convolution processing on the panoramic image to be processed to obtain a feature image of the panoramic image;

[0007] Performing regression processing on the feature image to obtain image transformation information of the panoramic image;

[0008] According to the image transformation information and the corrected pixel position of the panoramic image, obtaining a mapped pixel position corresponding to the corrected pixel position in the panoramic image; the corrected pixel position is represented as the coordinate position of a pixel in the corrected image corresponding to the panoramic image;

[0009] Performing pixel correction on the panoramic image according to the pixel information at the mapped pixel position to obtain a corrected image corresponding to the panoramic image.

[0010] In one embodiment, performing deformable convolution processing on the panoramic image to be processed to obtain a feature image of the panoramic image includes:

[0011] According to the sampling position offset of the panoramic image, performing offset processing on the sampling position of the panoramic image to obtain an offset sampling position of the panoramic image;

[0012] Performing linear interpolation processing on the offset sampling position to obtain an offset image corresponding to the panoramic image;

[0013] Performing convolution processing on the offset image to obtain a feature image of the panoramic image.

[0014] In one embodiment, before offsetting the sampling position of the panoramic image according to the sampling position offset of the panoramic image to obtain the offset sampling position of the panoramic image, it further includes:

[0015] Performing convolution processing on the panoramic image to obtain the sampling position offset of the panoramic image; the number of the sampling position offsets is equal to the number of convolution kernels in the convolution processing.

[0016] In one embodiment, performing regression processing on the feature image to obtain the image transformation information of the panoramic image includes:

[0017] Performing affine transformation processing on the feature image through a regression network to obtain the image transformation information of the panoramic image; the regression network is trained based on a sample panoramic image for the regression network to be trained.

[0018] In one embodiment, obtaining the mapped pixel position corresponding to the corrected pixel position in the panoramic image according to the image transformation information and the corrected pixel position of the panoramic image includes:

[0019] Multiplying the image transformation information and the corrected pixel position to obtain the mapped pixel position corresponding to the corrected pixel position in the panoramic image.

[0020] In one embodiment, performing pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain the corrected image corresponding to the panoramic image includes:

[0021] In the case where the coordinate value of the mapped pixel position is not an integer, performing linear interpolation processing on the coordinate value of the mapped pixel position to obtain a target pixel position; the coordinate value of the target pixel position is an integer;

[0022] Performing pixel correction on the panoramic image according to the pixel information of the target pixel position to obtain the corrected image corresponding to the panoramic image.

[0023] In one embodiment, performing pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain the corrected image corresponding to the panoramic image includes:

[0024] In the case where the coordinate value of the mapped pixel position is an integer, updating the pixel information of the corrected pixel position to the pixel information of the mapped pixel position corresponding to the corrected pixel position to obtain the corrected image corresponding to the panoramic image.

[0025] In one embodiment, the method further includes:

[0026] Perform image processing on the corrected image to obtain the target panoramic image corresponding to the corrected image.

[0027] In one embodiment, performing image processing on the corrected image to obtain the target panoramic image corresponding to the corrected image includes:

[0028] Perform super-resolution processing on the corrected image to obtain the target super-resolution panoramic image corresponding to the corrected image; the image resolution of the target super-resolution panoramic image is higher than the super-resolution image obtained based on the panoramic image to be processed.

[0029] Or,

[0030] Perform denoising processing on the corrected image to obtain the target denoised panoramic image corresponding to the corrected image; the image noise contained in the target denoised panoramic image is less than the denoised panoramic image obtained based on the panoramic image to be processed.

[0031] In a second aspect, the present application also provides a computer device. The computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0032] Perform deformable convolution processing on the panoramic image to be processed to obtain the feature image of the panoramic image;

[0033] Perform regression processing on the feature image to obtain the image transformation information of the panoramic image;

[0034] According to the image transformation information and the corrected pixel position of the panoramic image, obtain the mapped pixel position corresponding to the corrected pixel position in the panoramic image; the corrected pixel position is represented as the coordinate position of the pixel in the corrected image corresponding to the panoramic image.

[0035] According to the pixel information of the mapped pixel position, perform pixel correction on the panoramic image to obtain the corrected image corresponding to the panoramic image.

[0036] In a third aspect, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0037] Perform deformable convolution processing on the panoramic image to be processed to obtain the feature image of the panoramic image;

[0038] Perform regression processing on the feature image to obtain the image transformation information of the panoramic image;

[0039] Based on the image transformation information and the corrected pixel positions of the panoramic image, obtain the mapped pixel positions in the panoramic image corresponding to the corrected pixel positions; the corrected pixel positions are represented as the coordinate positions of the pixels in the corrected image corresponding to the panoramic image;

[0040] Perform pixel correction on the panoramic image according to the pixel information of the mapped pixel positions to obtain the corrected image corresponding to the panoramic image.

[0041] In a fourth aspect, the present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0042] Perform deformable convolution processing on the panoramic image to be processed to obtain a feature image of the panoramic image;

[0043] Perform regression processing on the feature image to obtain the image transformation information of the panoramic image;

[0044] Based on the image transformation information and the corrected pixel positions of the panoramic image, obtain the mapped pixel positions in the panoramic image corresponding to the corrected pixel positions; the corrected pixel positions are represented as the coordinate positions of the pixels in the corrected image corresponding to the panoramic image;

[0045] Perform pixel correction on the panoramic image according to the pixel information of the mapped pixel positions to obtain the corrected image corresponding to the panoramic image.

[0046] The above panoramic image processing method, computer device, storage medium, and computer program product perform deformable convolution processing on the panoramic image to be processed to obtain a feature image of the panoramic image, which can enhance the receptive field of the convolution process and extract more feature information from the panoramic image; then perform regression processing on the feature image to obtain the image transformation information of the panoramic image; based on the image transformation information and the corrected pixel positions of the panoramic image, obtain the mapped pixel positions in the panoramic image corresponding to the corrected pixel positions; the corrected pixel positions are represented as the coordinate positions of the pixels in the corrected image corresponding to the panoramic image, realizing the determination of the mapping relationship between the pixel positions of the panoramic image and the corrected pixel positions; perform pixel correction on the panoramic image according to the pixel information of the mapped pixel positions to obtain the corrected image corresponding to the panoramic image, solving the problem of imaging distortion of the panoramic image. Processing the panoramic image to be processed through the above panoramic image processing method is beneficial to improving the processing effect of other subsequent models on the panoramic image, and the panoramic image processing process can be flexibly combined with other models or networks, with a wide range of applications. Description of the Drawings

[0047] Figure 1Schematic flowchart of a panoramic image processing method in an embodiment;

[0048] Figure 2 Structural environment diagram of a panoramic image processing method in an embodiment;

[0049] Figure 3 Schematic flowchart of the steps for obtaining a feature image of a panoramic image in an embodiment;

[0050] Figure 4 Schematic flowchart of a panoramic image processing method in an embodiment;

[0051] Figure 5 Structural schematic diagram of a panoramic image processing method in another embodiment;

[0052] Figure 6 Flow environment diagram of a panoramic image processing method in an embodiment;

[0053] Figure 7 Internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer and more understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0055] In one embodiment, as Figure 1 shown, a panoramic image processing method is provided. In this embodiment, it is exemplified that the method is applied to a server. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0056] Step S101: Perform deformable convolution processing on the panoramic image to be processed to obtain a feature image of the panoramic image.

[0057] Step S102: Perform regression processing on the feature image to obtain image transformation information of the panoramic image;

[0058] Among them, the panoramic image to be processed refers to the panoramic image that needs to be processed. The panoramic image to be processed can be pre-stored in the server, or sent from the terminal to the server. Of course, it can also be obtained through other means. Among them, the panoramic image refers to an image of the scenery within a 360-degree spherical range. It should be noted that in each step of this method, in addition to using panoramic images, panoramic videos can also be used.

[0059] Among them, the image transformation information refers to the information of linear transformations such as image rotation, scaling, and translation of the panoramic image. The image transformation information includes, but is not limited to, the affine transformation information and the projection transformation information of the panoramic image.

[0060] Figure 2 is a structural schematic diagram of the above panoramic image processing method. As Figure 2 shown, in the above steps S101 and S102, the panoramic image is processed through the deformable localization network in the deformable spatial transformation model to obtain the image transformation information of the panoramic image. Specifically, the server obtains the panoramic image to be processed, performs convolution processing on the panoramic image to obtain the sampling position offset corresponding to the panoramic image; first adds the sampling position offset to the pixel positions of the panoramic image, and then uses the convolution kernel to perform multiple convolution processes on the superimposed panoramic image to obtain the feature image of the panoramic image. The server inputs the feature image into the regression network for regression processing; the regression network transforms the feature image to obtain the image transformation information (such as an image transformation matrix) of the panoramic image. Among them, the dimension of the image transformation matrix is determined according to the transformation type selected by the regression network for the feature image. Among them, the transformation type can be, but is not limited to, affine transformation and projection transformation. For example, if the regression network performs an affine transformation on the feature image, an image transformation matrix with a dimension of 2*3 can be obtained.

[0061] Furthermore, Figure 2 the deformable localization network in

[0062] Step S103, according to the image transformation information and the corrected pixel positions of the panoramic image, obtain the mapped pixel positions corresponding to the corrected pixel positions in the panoramic image; the corrected pixel positions are expressed as the coordinate positions of the pixels in the corrected image corresponding to the panoramic image.

[0063] Among them, the pixel positions of the panoramic image refer to the coordinates of each pixel in the panoramic image.

[0064] Among them, the corrected pixel positions refer to the coordinates of each pixel in the corrected image corresponding to the panoramic image. The corrected pixel positions are determined according to the size of the corrected image corresponding to the panoramic image set in advance.

[0065] As Figure 2As shown, in step S103, a mapping pixel position corresponding to the corrected pixel position in the panoramic image is obtained through the network generator (Gridgenerator) in the deformable spatial transformation model. Specifically, after the server obtains the size of the corrected image, it can determine the corrected pixel position in the corrected image, and then, based on the image transformation matrix, determine the mapping pixel position corresponding to the corrected pixel position in the panoramic image, that is, obtain the mapping relationship between the mapping pixel position and the corrected pixel position, so as to correct the pixel information at the mapping pixel position to the corrected pixel position in the subsequent steps. For example, if the corrected pixel position is (2, 4), and the server obtains, based on the image transformation matrix, that the mapping pixel position corresponding to the corrected pixel position (2, 4) in the panoramic image is (5, 6), a mapping relationship between the mapping pixel position (5, 6) and the corrected pixel position (2, 4) can be established.

[0066] Step S104: Pixel correction is performed on the panoramic image according to the pixel information at the mapping pixel position to obtain a corrected image corresponding to the panoramic image.

[0067] The corrected image refers to the image obtained after processing the panoramic image to be processed, and is used to replace the panoramic image to be processed as the processing object in the panoramic image processing method in the subsequent embodiments, so as to improve the processing effect of the panoramic image processing method.

[0068] As Figure 2 shown, in step S104, pixel correction is performed on the panoramic image through the sampler in the deformable spatial transformation model to obtain a corrected image corresponding to the panoramic image. Specifically, for the mapping relationship between the mapping pixel position and the corrected pixel position obtained in the above step S104, the sampler adds the pixel information at the mapping pixel position in the panoramic image to the corresponding corrected pixel position, thereby obtaining a corrected image corresponding to the panoramic image. For example, if there are image distortions in the people and animals in the panoramic image, the mapping pixel positions corresponding to the pixel positions of the corrected people and animals in the corrected image in the panoramic image can be determined first, and then the pixel information (such as color, transparency, etc.) at the mapping pixel position is added to the pixel information at the corrected pixel position, and finally a corrected panoramic image of the people and animals is obtained.

[0069] It should be noted that as Figure 2As shown in the figure, in this application, the traditional convolution in the localisation net of the Spatial transformer network is replaced with deformable convolution to construct a Deformable spatial transformer model. The Deformable spatial transformer model can not only increase the receptive field of the convolution kernel but also improve the adaptability to distorted objects in panoramic images. Therefore, this application can better adapt to different degrees of distortion at different positions in panoramic images. Among them, the distorted object in the panoramic image refers to the object with imaging distortion or deformation in the panoramic image. For example, in a panoramic image, the imaging of a wall may show distortions such as the wall becoming wider or the vertical wall bending, and then this wall can be regarded as a distorted object in the panoramic image.

[0070] In the above panoramic image processing method, the panoramic image to be processed is subjected to deformable convolution processing to obtain the feature image of the panoramic image, which can enhance the receptive field of the convolution process and extract more feature information from the panoramic image. Then, regression processing is performed on the feature image to obtain the image transformation information of the panoramic image. According to the image transformation information and the corrected pixel position of the panoramic image, the mapped pixel position corresponding to the corrected pixel position in the panoramic image is obtained. The corrected pixel position is represented as the coordinate position of the pixel in the corrected image corresponding to the panoramic image, and the mapping relationship between the pixel position of the panoramic image and the corrected pixel position is determined. According to the pixel information of the mapped pixel position, the panoramic image is pixel-corrected to obtain the corrected image corresponding to the panoramic image, and the problem of imaging distortion of the panoramic image is solved. By processing the panoramic image to be processed through the above panoramic image processing method, it is beneficial to improve the processing effect of subsequent other models on the panoramic image, and the panoramic image processing process can be flexibly combined with other models or networks, with a wide range of applications.

[0071] In one embodiment, as Figure 3 shown, in the above step S101, the panoramic image to be processed is subjected to deformable convolution processing to obtain the feature image of the panoramic image, which specifically includes the following steps:

[0072] Step S301: According to the sampling position offset of the panoramic image, the sampling position of the panoramic image is offset to obtain the offset sampling position of the panoramic image.

[0073] Among them, the sampling position offset refers to the offset direction information for each pixel in the panoramic image; the offset direction information represents the distance that the pixel offsets in the set direction, specifically including the x-axis direction and the y-axis direction; the sampling position offset is used to change the receptive field range of the convolution kernel without changing the convolution kernel.

[0074] Among them, the offset sampling position represents the position information of the sampling points for sampling the panoramic image.

[0075] After adding the learning of the sampling position offset in the Figure 2 Deformable localisation net, the size and position of the receptive field of the deformable convolution kernel can be dynamically adjusted according to the objects to be recognized in the panoramic image. For example, the receptive field of a traditional convolution kernel is generally in the form of 3*3, while the sampling position offset can change the receptive field of the convolution kernel from a 3*3 square to a shape and size similar to the objects to be recognized in the panoramic image. Specifically, the server performs standard convolution processing on the panoramic image to be processed to obtain the sampling position offset of the panoramic image; since the sampling position offset contains the offset direction information of each pixel in the panoramic image, the size of the sampling position offset is the same as the size of the panoramic image to be processed, and the sampling position offset can be superimposed on the pixel positions of each pixel of the panoramic image to obtain the offset sampling position of the panoramic image. In practical applications, the sampling position offset is superimposed on the distorted object in the panoramic image to obtain the offset sampling position of the distorted object, so that the convolution kernel can perform sampling processing on the offset sampling position of the distorted object, enabling the convolution kernel to collect more pixel information of the distorted object, thereby improving the processing effect on the distorted object in the panoramic image.

[0076] Step S302: Perform bilinear interpolation processing on the offset sampling position to obtain an offset image corresponding to the panoramic image.

[0077] Step S303: Perform convolution processing on the offset image to obtain a feature image of the panoramic image.

[0078] Specifically, the value of the offset sampling position of the panoramic image obtained in the above step S301 can be a non-integer, and the offset sampling position refers to the coordinate value of the pixel, and the offset sampling position does not contain the pixel information at the coordinate position in the image; furthermore, the server performs bilinear interpolation processing on the offset sampling position to obtain the target sampling position corresponding to the offset sampling position; according to the pixel information corresponding to the target sampling position, an offset image corresponding to the panoramic image is generated; a convolution kernel is used to perform convolution processing on the offset image to obtain a feature image of the panoramic image.

[0079] In this embodiment, by performing offset processing on the panoramic image through the sampling position offset, the receptive field of the ordinary convolution kernel can be enhanced, so as to extract more features of the distorted objects in the panoramic image, effectively improving the feature extraction effect of the panoramic image, which is beneficial to enhancing the processing effect on the distorted objects in the panoramic image in subsequent steps.

[0080] In one embodiment, before offsetting the sampling position of the panoramic image according to the sampling position offset of the panoramic image to obtain the offset sampling position of the panoramic image, the method further includes: performing convolution processing on the panoramic image to obtain the sampling position offset of the panoramic image; the number of sampling position offsets is equal to the number of convolution kernels in the convolution processing.

[0081] It should be noted that Figure 2 the second convolution processing of the deformable localization network can be multi-channel output. The number of convolution kernels in the second convolution processing is equal to the number of channels. It is necessary to calculate a sampling position offset for each convolution kernel in the second convolution processing, so as to use the sampling position offset to change the position of the pixels of the panoramic image input to the convolution kernel in the second convolution processing, so as to achieve the purpose of increasing the receptive field of the convolution kernel.

[0082] Specifically, the server performs standard convolution processing on the panoramic image to obtain the sampling position offset corresponding to the distorted object in the panoramic image. The sampling position offset can also be learned end-to-end through gradient backpropagation. During the training process of the deformable spatial transformation model in Figure 2 , the sampling position offset and the convolution kernels in the traditional convolution processing are updated simultaneously. Among them, since the sampling position offset contains the offset of each pixel of the panoramic image, the size of the sampling position offset is the same as the size of the input panoramic image, and the number of obtained sampling position offsets is equal to the number of convolution kernels in the convolution processing.

[0083] In this embodiment, by performing convolution processing on the panoramic image to obtain the sampling position offset, it is beneficial to enhance the receptive field of the convolution kernel through the sampling position offset in the subsequent steps, which helps to extract more features of the distorted object in the panoramic image.

[0084] In one embodiment, the above step S102 of performing regression processing on the feature image to obtain the image transformation information of the panoramic image specifically includes the following content: performing affine transformation processing on the feature image through a regression network to obtain the image transformation information of the panoramic image; the regression network is trained based on the sample panoramic image for the regression network to be trained.

[0085] Specifically, at the initial stage of training the regression network to be trained, the server performs an affine transformation on the sample panoramic image through the regression network to be trained, and an identity transformation matrix of the sample panoramic image can be obtained; then, according to the identity transformation matrix, the regression network to be trained is updated by gradient to obtain the regression network. Inputting the feature image into the regression network after gradient update again for affine transformation processing, an affine transformation matrix of the panoramic image can be obtained, that is, the image transformation information of the panoramic image. Among them, the affine transformation matrix contains linear transformation information such as rotation, scaling, and translation of the panoramic image.

[0086] It should be noted that although deformable convolution can increase the receptive field range of the convolution kernel, for panoramic images, its receptive field range is still not large enough. For example, when the panoramic image is at the extreme near both sides, the distortion degree will be particularly large, and the distance between relevant pixels may be 100 pixels or even more than 100 pixels. The sampling position offset cannot make the receptive field cover the relevant pixels. Therefore, it is necessary to globally process the feature image through the regression network to obtain the image transformation information of the panoramic image.

[0087] In this embodiment, by performing affine transformation on the feature image through the regression network, the defect that the receptive field of deformable convolution cannot cover the global distortion in the panoramic image can be made up for, so that the image transformation information obtained in this embodiment can comprehensively reflect the distortion at different positions and degrees in the panoramic image, which helps to improve the processing effect of the panoramic image.

[0088] In one embodiment, in step S103 above, according to the image transformation information and the corrected pixel position of the panoramic image, the mapped pixel position corresponding to the corrected pixel position of the panoramic image is obtained, which specifically includes the following content: multiplying the image transformation information and the corrected pixel position to obtain the mapped pixel position corresponding to the corrected pixel position in the panoramic image.

[0089] Specifically, the server performs matrix operation on the image transformation information and the corrected pixel position of the panoramic image to obtain the mapped pixel position corresponding to the corrected pixel position in the panoramic image; establish a mapping relationship between the corrected pixel position and the mapped pixel position. The obtained mapped pixel position corresponding to the corrected pixel position in the panoramic image can be represented by the following formula:

[0090]

[0091] where x v and y v respectively represent the x-axis coordinate and y-axis coordinate of the corrected pixel position; θ 11 、θ 12 、θ 13 、θ 21 、θ 22 and θ23 representing the respective image transformation information corresponding to the panoramic image in the image transformation matrix; x u and y u respectively represent the x-axis coordinate and y-axis coordinate of the mapped pixel position corresponding to the corrected pixel position.

[0092] In this embodiment, by means of the image transformation information of the panoramic image output by the Deformable localisation net and the corrected pixel positions of the panoramic image, it is possible to establish a one-to-one correspondence between the corrected pixel positions and the mapped pixel positions in the panoramic image, so as to facilitate the subsequent steps of correcting the panoramic image based on the mapped pixel positions.

[0093] In one embodiment, in step S104 above, according to the pixel information of the mapped pixel positions, the panoramic image is pixel-corrected to obtain a corrected image corresponding to the panoramic image, which specifically includes the following: when the coordinate values of the mapped pixel positions are not integers, linear interpolation processing is performed on the coordinate values of the mapped pixel positions to obtain target pixel positions; the coordinate values of the target pixel positions are integers; according to the pixel information of the target pixel positions, the panoramic image is pixel-corrected to obtain a corrected image corresponding to the panoramic image.

[0094] Specifically, the coordinate values of the mapped pixel positions corresponding to the corrected pixel positions determined by the server according to the image transformation information may not be integers. When the coordinate values of the mapped pixel positions corresponding to the corrected pixel positions are not integers, it is impossible to directly obtain the corresponding pixel information in the panoramic image, and thus the server is also unable to determine the pixel information to be placed at the corrected pixel positions. Therefore, the server needs to perform bilinear interpolation processing on the coordinate values of the mapped pixel positions corresponding to the corrected pixel positions to obtain target pixel positions with integer coordinate values.

[0095] Furthermore, the server updates the mapped pixel positions corresponding to the corrected pixel positions to the target pixel positions, obtaining the mapping relationship between the corrected pixel positions and the target pixel positions of the panoramic image. According to the pixel information of the target pixel positions, the panoramic image is pixel-corrected to obtain a corrected image corresponding to the panoramic image.

[0096] For example, the corrected pixel position is (3, 4), and the pixel position of the panoramic image corresponding to the corrected pixel position is (1.2, 5.6). Since the pixel positions of the panoramic image are usually integers, it is impossible to find the pixel information with the pixel position of (1.2, 5.6) in the panoramic image, and the subsequent steps cannot be executed to obtain the corrected image. Suppose the target pixel position obtained after bilinear interpolation processing is (1, 6). Then, based on the pixel information of the target pixel position (1, 6) in the panoramic image, the pixel information of the corrected pixel position (3, 4) is updated to obtain the corrected image corresponding to the panoramic image.

[0097] In this embodiment, when the coordinate value of the mapped pixel position is not an integer, linear interpolation processing is performed on the coordinate value of the mapped pixel position to obtain the target pixel position; then, based on the pixel information of the target pixel position, pixel correction is performed on the panoramic image to obtain the corrected image corresponding to the panoramic image, thereby determining the mapping relationship between the target pixel position of the panoramic image and the corrected pixel position in the corrected image, which is beneficial to updating the pixel information of the target pixel position of the panoramic image to the corrected pixel position of the corrected image.

[0098] In one embodiment, step S104 above, performing pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain the corrected image corresponding to the panoramic image, specifically includes the following content: when the coordinate value of the mapped pixel position is an integer, the pixel information of the corrected pixel position is updated to the pixel information of the mapped pixel position corresponding to the corrected pixel position to obtain the corrected image corresponding to the panoramic image.

[0099] Specifically, the server collects the pixel information of the mapped pixel position or the target pixel position of the panoramic image through a sampler, and then fills the pixel information into the corresponding corrected pixel position; after pixel information is obtained at all corrected pixel positions, the corrected image corresponding to the panoramic image to be processed is obtained.

[0100] In this embodiment, by updating the pixel information of the corrected pixel position to the pixel information of the mapped pixel position corresponding to the corrected pixel position, the corrected image corresponding to the panoramic image to be processed is obtained, solving the problem of imaging distortion of the panoramic image to be processed and realizing the correction processing of the distorted imaging in the panoramic image to be processed.

[0101] In one embodiment, after performing pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain the corrected image corresponding to the panoramic image, the following content is further included: performing image processing on the corrected image to obtain the target panoramic image corresponding to the corrected image.

[0102] In this embodiment, by first processing the panoramic image to be processed, a corrected image corresponding to the panoramic image to be processed can be obtained, and then the image processing model is allowed to perform image processing on the corrected image. Without changing the original structure of the image processing model, the processing effect of the image processing model on the corrected image can be improved, and the quality of the obtained target panoramic image is greatly improved.

[0103] In one embodiment, as Figure 4 shown, a panoramic image processing method is provided. In this embodiment, taking the application of this method to a server as an example, it can be understood that this method can also be applied to a terminal, or to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0104] Step S401, obtain a corrected image corresponding to the panoramic image to be processed.

[0105] Among them, the corrected image is realized through the above steps S101 to S104, which will not be elaborated here.

[0106] Step S402, perform image processing on the corrected image to obtain a target panoramic image corresponding to the corrected image.

[0107] Among them, the image processing can be super-resolution processing, denoising processing, or frame interpolation processing. The specific image processing method can be flexibly changed according to requirements and will not be specifically limited here.

[0108] Specifically, Figure 5 is a schematic diagram of the application environment of the above panoramic image processing method. As Figure 5 shown, the server obtains the panoramic image to be processed; before performing image processing on the panoramic image to be processed, first process the panoramic image to be processed through the above panoramic image processing method to obtain a corrected image corresponding to the panoramic image to be processed. Then the server inputs the corrected image into other image processing models or networks for image processing to obtain a target panoramic image corresponding to the corrected image.

[0109] In this embodiment, due to the distorted imaging of the panoramic image, the image processing effect of the image processing model is low when processing the panoramic image. By processing the panoramic image to be processed through the above panoramic image processing method, a corrected image corresponding to the panoramic image to be processed can be obtained, and the image processing model is allowed to perform image processing on the corrected image. Without changing the original structure of the image processing model, the processing effect of the image processing model can be improved, and the quality of the obtained target panoramic image is greatly improved.

[0110] In one embodiment, image processing is performed on the corrected image to obtain a target panoramic image corresponding to the corrected image, which specifically includes the following: performing super-resolution processing on the corrected image to obtain a target super-resolution panoramic image corresponding to the corrected image; the image resolution of the target super-resolution panoramic image is higher than that of the super-resolution image obtained based on the panoramic image to be processed; or, performing denoising processing on the corrected image to obtain a target denoised panoramic image corresponding to the corrected image; the image noise included in the target denoised panoramic image is less than that of the denoised panoramic image obtained based on the panoramic image to be processed.

[0111] It should be noted that the corrected image obtained through the above steps S101 to S104 can be input into various types of image processing models for image processing. For example, a super-resolution model, a denoising model, and an interpolation model.

[0112] Specifically, the corrected image can be input into a super-resolution model for super-resolution processing to obtain a target super-resolution panoramic image corresponding to the corrected image, so as to improve the resolution of the corrected image; since the corrected image has been pre-corrected, the super-resolution model can perform super-resolution processing on each object in the corrected image more accurately, while the distorted objects in the panoramic image to be processed will affect the processing effect of the super-resolution model, so the image resolution of the target super-resolution panoramic image is higher than that of the super-resolution image obtained based on the panoramic image to be processed. Or the corrected image can be input into a denoising model for denoising processing to obtain a target denoised panoramic image corresponding to the corrected image, so as to reduce the noise in the panoramic image; since the corrected image has been pre-corrected, the super-resolution model can perform denoising processing on the corrected image more accurately, while the distorted objects in the panoramic image to be processed are easily misrecognized as noise points by the denoising model, resulting in a worse denoising effect of the denoising model on the panoramic image to be processed than on the corrected image, so the image noise included in the target denoised panoramic image is less than that of the denoised panoramic image obtained based on the panoramic image to be processed.

[0113] In this embodiment, the corrected image obtained through the above panoramic image processing method can not only be combined with various types of image processing models for use, but also does not need to change the original structure of the image processing model, and can effectively improve the quality of the target panoramic image processed by the image processing model in a scenario with limited computing power, making the panoramic image processing method in this embodiment have a wide range of applications and a good effect in processing panoramic images.

[0114] In one embodiment, as Figure 6 shown, another panoramic image processing method is provided. Taking the application of this method to a server as an example for illustration, it includes the following steps:

[0115] Step S601: Perform convolution processing on the panoramic image to be processed to obtain the sampling position offset of the panoramic image; the number of sampling position offsets is equal to the number of convolution kernels in the convolution processing.

[0116] Step S602: According to the sampling position offset of the panoramic image, perform offset processing on the sampling position of the panoramic image to obtain the offset sampling position of the panoramic image.

[0117] Step S603: Perform linear interpolation processing on the offset sampling position to obtain the offset image corresponding to the panoramic image; perform convolution processing on the offset image to obtain the feature image of the panoramic image.

[0118] Step S604: Perform affine transformation processing on the feature image through a regression network to obtain the image transformation information of the panoramic image; the regression network is trained based on the sample panoramic image for the regression network to be trained.

[0119] Step S605: Multiply the image transformation information and the corrected pixel position to obtain the mapped pixel position corresponding to the corrected pixel position in the panoramic image; the corrected pixel position is represented as the coordinate position of the pixel in the corrected image corresponding to the panoramic image. It can be understood that the mapped pixel position refers to the coordinate position of the pixel in the panoramic image, and the corrected pixel position refers to the coordinate position of the pixel in the corrected image, and there is a mapping relationship between the coordinate positions of the two types of pixels.

[0120] Step S606-1: In the case where the coordinate value of the mapped pixel position is not an integer, update the pixel information of the corrected pixel position to the pixel information of the mapped pixel position corresponding to the corrected pixel position to obtain the corrected image corresponding to the panoramic image.

[0121] Step S606-2: In the case where the coordinate value of the mapped pixel position is not an integer, perform linear interpolation processing on the coordinate value of the mapped pixel position to obtain the target pixel position; the coordinate value of the target pixel position is an integer; according to the pixel information of the target pixel position, perform pixel correction on the panoramic image to obtain the corrected image corresponding to the panoramic image.

[0122] It can be understood that before Step S606-1 and Step S606-2, it is possible to determine whether the coordinate value of the mapped pixel position in the panoramic image is an integer, so as to determine whether to execute Step S606-1 or Step S606-2 according to the judgment result.

[0123] The above panoramic image processing method has the following beneficial effects: performing deformable convolution processing on the panoramic image to be processed to obtain a feature image of the panoramic image, which can enhance the receptive field of the convolution process and extract more feature information from the panoramic image; then performing regression processing on the feature image to obtain image transformation information of the panoramic image; according to the image transformation information and the corrected pixel position of the panoramic image, obtaining the mapped pixel position corresponding to the corrected pixel position in the panoramic image; the corrected pixel position is represented as the coordinate position of the pixel in the corrected image corresponding to the panoramic image, realizing the determination of the mapping relationship between the pixel position of the panoramic image and the corrected pixel position; according to the pixel information of the mapped pixel position, performing pixel correction on the panoramic image to obtain the corrected image corresponding to the panoramic image, solving the problem of imaging distortion of the panoramic image. Processing the panoramic image to be processed by the above panoramic image processing method is beneficial to improving the processing effect of other models on the panoramic image, and the panoramic image processing process can be flexibly combined with other models or networks, with a wide range of applications.

[0124] To more clearly illustrate the panoramic image processing method provided by the embodiments of the present disclosure, the following uses a specific embodiment to specifically describe the panoramic image processing method. In one embodiment, another panoramic image processing method is provided, which can be applied to an application environment as shown in Figure 2 and specifically includes the following content:

[0125] First, perform convolution processing on the panoramic image to be processed through a convolution layer to obtain the sampling position offset of the panoramic image to be processed, add the learned sampling position offset to the original input panoramic image to be processed, and obtain the offset image of the panoramic image to be processed through bilinear interpolation. Then, perform ordinary convolution on the offset image to obtain a feature image; perform matrix transformation on the feature image through a regression network to obtain an image transformation matrix; while increasing the receptive field of the convolution kernel through deformable convolution, the adaptability to different positions and different degrees of distortion in the panoramic image can be improved through matrix transformation. Among them, the image transformation matrix is an identity transformation matrix at the beginning, and after experiencing training and learning gradient update, the image transformation matrix will become an affine transformation matrix, which contains all linear transformation information such as rotation, scaling, and translation of the panoramic image. Perform matrix operation on the image transformation matrix of the panoramic image and the corrected pixel position of the panoramic image to obtain the mapped pixel position corresponding to the corrected pixel position in the panoramic image. Finally, fill the pixel information of the mapped pixel position into the pixel information of the corresponding corrected pixel position to obtain the corrected image corresponding to the panoramic image.

[0126] In this embodiment, in view of the characteristic of panoramic image distortion, a panoramic image processing method is proposed, which can be widely used before any model or network processes panoramic images, so as to improve the image processing effect of the model or network without modifying the original model or network. It only needs to apply this panoramic image processing method before processing panoramic images, which is convenient to operate and can also bring about an improvement in the effect, with a wide range of applications.

[0127] It should be understood that although the steps in the flowcharts involved in the above-described embodiments are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless there is a clear indication in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-described embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0128] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as Figure 7 shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as panoramic images to be processed, corrected images, and target panoramic images. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a panoramic image processing method or a panoramic image processing method.

[0129] Those skilled in the art can understand that Figure 7 the structure shown in

[0130] is only a block diagram of a part of the structure related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements. In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the above method embodiments are implemented.

[0131] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the foregoing method embodiments are implemented.

[0132] In one embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the foregoing method embodiments are implemented.

[0133] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, image data, etc.) involved in this application are all information and data that have been authorized by the user or fully authorized by all parties.

[0134] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0135] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0136] The above-described embodiments merely represent several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A panoramic image processing method, characterized in that, The method includes: Performing deformable convolution processing on the panoramic image to be processed through a deformable spatial transformation model to obtain a feature image of the panoramic image; the deformable spatial transformation model is obtained by replacing the convolution of the positioning network in the spatial transformation network with deformable convolution; Performing regression processing on the feature image to obtain image transformation information of the panoramic image; the regression processing is used to perform global processing on the feature image; the image transformation information is used to reflect the distortion at different positions and degrees in the panoramic image; According to the image transformation information and the corrected pixel position of the panoramic image, obtaining a mapped pixel position corresponding to the corrected pixel position in the panoramic image; the corrected pixel position is represented as the coordinate position of the pixel in the corrected image corresponding to the panoramic image; Performing pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain a corrected image corresponding to the panoramic image.

2. The method according to claim 1, wherein The performing deformable convolution processing on the panoramic image to be processed to obtain a feature image of the panoramic image includes: According to the sampling position offset of the panoramic image, performing offset processing on the sampling position of the panoramic image to obtain an offset sampling position of the panoramic image; Performing linear interpolation processing on the offset sampling position to obtain an offset image corresponding to the panoramic image; Performing convolution processing on the offset image to obtain a feature image of the panoramic image.

3. The method according to claim 2, wherein Before performing offset processing on the sampling position of the panoramic image according to the sampling position offset of the panoramic image to obtain an offset sampling position of the panoramic image, it further includes: Performing convolution processing on the panoramic image to obtain the sampling position offset of the panoramic image; the number of the sampling position offsets is equal to the number of convolution kernels in the convolution processing.

4. The method according to claim 1, wherein The performing regression processing on the feature image to obtain image transformation information of the panoramic image includes: Performing affine transformation processing on the feature image through a regression network to obtain image transformation information of the panoramic image; the regression network is trained based on a sample panoramic image for the regression network to be trained.

5. The method according to claim 1, characterized in that, The obtaining a mapped pixel position corresponding to the corrected pixel position in the panoramic image according to the image transformation information and the corrected pixel position of the panoramic image includes: Multiplying the image transformation information and the corrected pixel position to obtain a mapped pixel position corresponding to the corrected pixel position in the panoramic image.

6. The method according to claim 1, characterized in that The performing pixel correction on the panoramic image according to the pixel information of the mapped pixel position to obtain a corrected image corresponding to the panoramic image includes: In the case where the coordinate value of the mapped pixel position is not an integer, performing linear interpolation processing on the coordinate value of the mapped pixel position to obtain a target pixel position; the coordinate value of the target pixel position is an integer; Performing pixel correction on the panoramic image according to the pixel information of the target pixel position to obtain a corrected image corresponding to the panoramic image.

7. The method according to claim 1, characterized in that Performing pixel correction on the panoramic image according to the pixel information at the mapped pixel positions to obtain a corrected image corresponding to the panoramic image, including: When the coordinate values of the mapped pixel positions are integers, updating the pixel information at the corrected pixel positions to the pixel information at the mapped pixel positions corresponding to the corrected pixel positions to obtain a corrected image corresponding to the panoramic image.

8. The method according to any one of claims 1-7, characterized in that, Further including: Performing image processing on the corrected image to obtain a target panoramic image corresponding to the corrected image.

9. The method according to claim 8, wherein The performing image processing on the corrected image to obtain a target panoramic image corresponding to the corrected image includes: Performing super-resolution processing on the corrected image to obtain a target super-resolution panoramic image; the image resolution of the target super-resolution panoramic image is higher than that of the super-resolution image obtained based on the panoramic image to be processed; Or, Performing denoising processing on the corrected image to obtain a target denoised panoramic image; the image noise included in the target denoised panoramic image is less than that of the denoised panoramic image obtained based on the panoramic image to be processed.

10. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer program product, comprising a computer program, characterized in that, When this computer program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 9.

Images