Panoramic image and panoramic video super-resolution processing method, device and product

Abstract

The application relates to the technical field of image processing, and provides a panoramic image and panoramic video super-resolution processing method, device and product, which can make the panoramic image clearer while retaining more image details, and effectively optimize the super-resolution reconstruction effect. The method comprises the following steps: acquiring a panoramic image, and obtaining a pre-trained reversible neural network model; the reversible neural network model comprises a deformation removal network and a deformation restoration network corresponding to the deformation removal network; the panoramic image is input into the reversible neural network model, the panoramic image is converted into a deformation-removed panoramic image by the deformation removal network in the reversible neural network model, the deformation-removed panoramic image is output to a super-resolution network, and a super-resolution image of the deformation-removed panoramic image is obtained by the super-resolution network; and the super-resolution image of the deformation-removed panoramic image is input into the deformation restoration network, and the super-resolution image of the deformation-removed panoramic image is converted into a super-resolution image of the panoramic image by the deformation restoration network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a panoramic image super-resolution processing method, a panoramic video super-resolution processing method, a computer device and a computer program product. BACKGROUND

[0002] With the development of computer technology, panoramic image shooting is increasingly popular. A panoramic image is an image taken by a specified shooting device at multiple angles. Due to the constraints of device performance, although the panoramic image has a wider field of view and richer scenes, there is a situation of image blur.

[0003] In related technologies, a panoramic image can be processed by a super-resolution processing model, but the processing effect is poor and there is a problem of too much loss of details of the panoramic image. SUMMARY

[0004] Therefore, it is necessary to provide a panoramic image super-resolution processing method, a panoramic video super-resolution processing method, a computer device and a computer program product to solve the above technical problems.

[0005] In a first aspect, the present application provides a panoramic image super-resolution processing method. The method comprises:

[0006] obtaining a panoramic image, and obtaining a pre-trained reversible neural network model; the reversible neural network model comprises a deforming network and a corresponding deforming restoration network of the deforming network;

[0007] inputting the panoramic image into the reversible neural network model, converting the panoramic image into a deformed panoramic image by the deforming network in the reversible neural network model, and outputting the deformed panoramic image to a super-resolution network, and obtaining a super-resolution image of the deformed panoramic image by the super-resolution network;

[0008] inputting the super-resolution image of the deformed panoramic image into the deforming restoration network, and converting the super-resolution image of the deformed panoramic image into a super-resolution image of the panoramic image by the deforming restoration network.

[0009] In a second aspect, the present application further provides a panoramic video super-resolution processing method. The method comprises:

[0010] after receiving a panoramic video sent by a terminal, frame the panoramic video to obtain a plurality of panoramic images;

[0011] According to any one of the panoramic image super-resolution processing methods, obtain a super-resolution image of each panoramic image in the plurality of panoramic images;

[0012] According to the super-resolution image of each frame of the plurality of panoramic images, a super-resolution panoramic video of the panoramic video is obtained.

[0013] The super-resolution panoramic video of the panoramic video is sent to the terminal.

[0014] In a third aspect, the present application also provides a computer device. The computer device comprises a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0015] A panoramic image is obtained, and a pre-trained reversible neural network model is obtained. The reversible neural network model comprises a deforming network and a corresponding deforming restoration network of the deforming network.

[0016] The panoramic image is input into the reversible neural network model. The deforming network in the reversible neural network model converts the panoramic image into a deformed panoramic image. The deformed panoramic image is output to a super-resolution network. The super-resolution network obtains a super-resolution image of the deformed panoramic image.

[0017] The super-resolution image of the deformed panoramic image is input into the deforming restoration network. The deforming restoration network converts the super-resolution image of the deformed panoramic image into a super-resolution image of the panoramic image.

[0018] In a fourth aspect, the present application also provides a computer device. The computer device comprises a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0019] After receiving the panoramic video sent by the terminal, the panoramic video is framed to obtain a plurality of panoramic images.

[0020] According to any one of the panoramic image super-resolution processing methods, a super-resolution image of each frame of the plurality of panoramic images is obtained.

[0021] According to the super-resolution image of each frame of the plurality of panoramic images, a super-resolution panoramic video of the panoramic video is obtained.

[0022] The super-resolution panoramic video of the panoramic video is sent to the terminal.

[0023] In a fifth aspect, the present application also provides a computer program product. The computer program product comprises a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0024] A panoramic image is acquired, and a pre-trained reversible neural network model is obtained; the reversible neural network model includes a deformation removal network and a deformation restoration network corresponding to the deformation removal network.

[0025] The panoramic image is input into a reversible neural network model, where the deformation removal network in the reversible neural network model converts the panoramic image into a deformation removal panoramic image, and the deformation removal panoramic image is output to a super-resolution network, where the super-resolution network obtains a super-resolution image of the deformation removal panoramic image.

[0026] The super-resolution image of the de-distorted panoramic image is input into the deformation restoration network, which then converts the super-resolution image of the de-distorted panoramic image into the super-resolution image of the panoramic image.

[0027] Sixthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0028] After receiving the panoramic video sent by the terminal, the panoramic video is divided into frames to obtain multiple panoramic images;

[0029] According to the panoramic image super-resolution processing method described above, obtain the super-resolution image of each frame of the panoramic image in the multi-frame panoramic image.

[0030] The super-resolution panoramic video of the panoramic video is obtained based on the super-resolution image of each panoramic image in the multi-frame panoramic image.

[0031] The super-resolution panoramic video of the panoramic video is sent to the terminal.

[0032] The panoramic image super-resolution processing method, the panoramic video super-resolution processing method, the computer device, and the computer program product can obtain a panoramic image, and obtain a pre-trained reversible neural network model, the reversible neural network model comprising a deforming network and a deforming reduction network corresponding to the deforming network; then the panoramic image is input into the reversible neural network model, the deforming network in the reversible neural network model converts the panoramic image into a deformed panoramic image, and the deformed panoramic image is output to a super-resolution network, the super-resolution network obtains a super-resolution image of the deformed panoramic image, the super-resolution image of the deformed panoramic image is input into the deforming reduction network, and then the deforming reduction network converts the super-resolution image of the deformed panoramic image into a super-resolution image of the panoramic image. The embodiment can make the panoramic image to be processed by the super-resolution network more matched by removing the image deformation in the panoramic image, avoid the original image deformation in the panoramic image as interference to affect the processing of the super-resolution network, make the finally obtained super-resolution image more clear while retaining more image details, and eliminate image noise, effectively optimizing the super-resolution reconstruction effect. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 A flowchart of a panoramic image super-resolution processing method in an embodiment;

[0034] Figure 2 A panoramic image sample in an embodiment;

[0035] Figure 3-a A deformed panoramic image in an embodiment;

[0036] Figure 3-b Another deformed panoramic image in an embodiment;

[0037] Figure 4 A flowchart of a training super-resolution network in an embodiment;

[0038] Figure 5 An application environment diagram of a panoramic video super-resolution processing method in an embodiment;

[0039] Figure 6 A flowchart of a panoramic video super-resolution processing method in an embodiment;

[0040] Figure 7 A flowchart of another panoramic video super-resolution processing method in an embodiment;

[0041] Figure 8 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0042] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application and not to limit the present application.

[0043] In one embodiment, as shown in Figure 1 A panoramic image super-resolution processing method is provided. The method can be applied to a server, a terminal, or a system including a terminal and a server, and can be implemented through the interaction between the terminal and the server. The method includes the following steps:

[0044] S110, obtaining a panoramic image and a pre-trained reversible neural network model; the reversible neural network model includes a deformation removal network and a deformation restoration network corresponding to the deformation removal network.

[0045] As an example, the panoramic image can be an image captured using an angle greater than a preset angle threshold or an image rendered using modeling software. The panoramic image has imaging distortion, and one or more photographed objects are deformed in the panoramic image.

[0046] In actual applications, a panoramic image can be obtained. The panoramic image can be generated by a professional shooting device (such as a VR panoramic camera) capturing image information of an entire scene or by a modeling software. Compared with a traditional shooting device (such as a camera with a standard lens), the panoramic image can cover the sides, top, bottom, and back that cannot be captured by the traditional shooting device. However, the panoramic image has imaging distortion due to the wide angle.

[0047] In addition, a pre-trained reversible neural network model can be obtained. The pre-trained reversible neural network model can include a deformation removal network and a deformation restoration network corresponding to the deformation removal network. The deformation removal network can remove the deformation of an image input into the deformation removal network, and the deformation restoration network can restore the deformation removed by the deformation removal network to the image input into the deformation restoration network.

[0048] S120, inputting the panoramic image into the reversible neural network model, converting the panoramic image into a deformation-removed panoramic image by the deformation removal network in the reversible neural network model, and outputting the deformation-removed panoramic image to a super-resolution network to obtain a super-resolution image of the deformation-removed panoramic image by the super-resolution network.

[0049] As an example, the deformation-removed panoramic image can be an image in which the deformation of the photographed object in the panoramic image is removed, and the deformation-removed panoramic image can correctly reflect the shape of the photographed object.

[0050] After obtaining the panoramic image and the trained reversible neural network model, the panoramic image can be input into the reversible neural network model, the image deformation existing in the panoramic image is removed by the deformation removal network in the reversible neural network model, and a deformation-removed panoramic image without image deformation is converted, and then the deformation-removed panoramic image can be output to the super-resolution network, and the super-resolution network is used for super-resolution processing on the deformation-removed panoramic image, so as to obtain a super-resolution image of the deformation-removed panoramic image, that is, a deformation-removed panoramic image with higher resolution can be obtained.

[0051] S130, inputting the super-resolution image of the deformation-removed panoramic image into the deformation restoration network, and converting the super-resolution image of the deformation-removed panoramic image into a super-resolution image of the panoramic image by the deformation restoration network.

[0052] Specifically, after obtaining the super-resolution image of the deformation-removed panoramic image, the super-resolution image can be input into the deformation restoration network, and the deformation content removed from the panoramic image is restored to the super-resolution image of the deformation-removed panoramic image again, so as to obtain a super-resolution image of the panoramic image, which improves the resolution of the panoramic image while retaining the image deformation effect of the panoramic image.

[0053] In this embodiment, a panoramic image can be obtained, and a reversible neural network model trained in advance is obtained, the reversible neural network model includes a deformation removal network and a deformation restoration network corresponding to the deformation removal network; then the panoramic image can be input into the reversible neural network model, the panoramic image is converted into a deformation-removed panoramic image by the deformation removal network in the reversible neural network model, and the deformation-removed panoramic image is output to a super-resolution network, the super-resolution network obtains a super-resolution image of the deformation-removed panoramic image, the super-resolution image of the deformation-removed panoramic image is input into a deformation restoration network, and then the super-resolution image of the deformation-removed panoramic image is converted into a super-resolution image of the panoramic image by the deformation restoration network. Through removing the image deformation in the panoramic image, the panoramic image to be processed by the super-resolution network is more matched with the super-resolution network in this embodiment, the original image deformation in the panoramic image is avoided as interference to affect the processing of the super-resolution network, so that the finally obtained super-resolution image is clearer while retaining more image details, and the image noise is eliminated, and the super-resolution reconstruction effect is effectively optimized.

[0054] In one embodiment, the method can further include:

[0055] S210, obtaining a panoramic image sample and a deformation-removed panoramic image of the panoramic image sample.

[0056] In a specific implementation, a panoramic image sample for training the reversible neural network model can be obtained, the panoramic image sample being a panoramic image with one or more image distortions, and a deformed panoramic image of the panoramic image sample can be obtained, wherein the deformed panoramic image can be obtained by removing the image deformation in the panoramic image sample. In an exemplary embodiment, obtaining the deformed panoramic image of the panoramic image sample can include the following steps:

[0057] The panoramic image sample in the form of image projection is converted into an image in the form of image projection of a cube to obtain the deformed panoramic image of the panoramic image sample.

[0058] In actual applications, the image projection of the panoramic image sample is in the form of a sphere, and image distortion such as line bending is prone to occur in the panoramic image sample, for example, Figure 2 In the illustrated panoramic image sample, the steps in the white dashed line frame and the outer frame of the power line in the black dashed line frame are both curved to some extent.

[0059] After obtaining the panoramic image sample in the form of image projection of a sphere, the image projection of the panoramic image sample can be converted from a sphere to a cube, and the image after the image projection conversion can be used as the deformed panoramic image of the panoramic image sample. In an example, a special image processing software (such as Pano2VR software) developed for panoramic pictures can be used to convert the projection of the panoramic image sample from a rectangular spherical projection to a cubic projection.

[0060] Exemplarily, after conversion, Figure 2 The image content in the white dashed line frame can be as shown in Figure 3-a The curved lines of the steps become straight lines, and Figure 2 The image content in the black dashed line frame can also be as shown in Figure 3-b It can be seen that the degree of distortion of the image after the image projection conversion is greatly reduced.

[0061] By converting the panoramic image sample in the form of image projection of a sphere into an image in the form of image projection of a cube, the deformed panoramic image of the panoramic image sample can be quickly obtained, and the data pair for training the reversible neural network model can be efficiently generated.

[0062] S220, inputting the panoramic image sample into the reversible neural network model to be trained, outputting the predicted deformed panoramic image from the deformed network in the reversible neural network model to be trained to the super-resolution network, so that the super-resolution network outputs the super-resolution image of the predicted deformed panoramic image.

[0063] After obtaining the panoramic image sample, the panoramic image sample can be input to the reversible neural network model to be trained. In actual application, the network basic unit of the reversible neural network model is a reversible block composed of two complementary affine coupling layers, and the data processing framework is that when the model processes input data, the input data can be divided into u1 and u2, and converted by learning functions s and t and coupled in an alternating manner. Specifically, in the forward process, u1 and u2 can obtain v1 and v2 through the following transformations:

[0064] v1=u1⊙exp(s2(u2))+t2(u2)

[0065] v2=u1⊙exp(s1(v1))+t1(v1)

[0066] Where s1, t1, s2 and t2 are learning functions. The forward process of the reversible neural network model can obtain the final output v based on v1 and v2. Given the output v, the reversible process of the above formula can be obtained by dividing v into v1 and v2, and then transforming v1 and v2 to obtain u1 and u2 as follows:

[0067] u2=(v2-t1(v1))⊙exp(-s1(v1))

[0068] u1=(v1-t2(u2))⊙exp(-s2(u2))

[0069] In this embodiment, the reversible neural network model to be trained includes a deforming network and a deforming restoration network. For the currently input panoramic image sample, the deforming network in the reversible neural network model to be trained can output a predicted deforming panoramic image to the super-resolution network, and the super-resolution network can further perform super-resolution processing on the predicted deforming panoramic image to improve the resolution of the input image and obtain a super-resolution image of the predicted deforming panoramic image output by the super-resolution network.

[0070] S230, input the super-resolution image of the predicted deforming panoramic image to the reversible neural network model to be trained, and output a super-resolution image of the predicted panoramic image sample by the deforming restoration network in the reversible neural network model to be trained.

[0071] Specifically, the super-resolution image of the predicted deforming panoramic image can be input to the reversible neural network model to be trained, and the deforming restoration network in the reversible neural network model to be trained can perform deforming processing on one or more contents in the super-resolution image of the predicted deforming panoramic image, so that the super-resolution image of the predicted deforming panoramic image is distorted, and then a super-resolution image of the predicted panoramic image sample output by the deforming restoration network can be obtained.

[0072] S240, determining the model loss of the reversible neural network model to be trained according to the panorama image sample, the deformed panorama image of the panorama image sample, the predicted deformed panorama image, and the super-resolution image of the predicted panorama image sample.

[0073] After obtaining the super-resolution image of the predicted panorama image sample, the model loss of the reversible neural network model to be trained can be determined according to the panorama image sample, the deformed panorama image of the panorama image sample, and the predicted deformed panorama image output by the reversible neural network model to be trained and the super-resolution image of the predicted panorama image sample.

[0074] In an embodiment, the model loss of the reversible neural network model to be trained can include a first loss, a second loss, and a third loss, and step S240 can include the following steps:

[0075] The first loss is obtained according to the panorama image sample and the super-resolution image of the predicted panorama image sample; the second loss is obtained according to the correlation between the predicted deformed panorama image and the preset deformed content; the third loss is obtained according to the deformed panorama image of the panorama image sample and the predicted deformed panorama image; and the model loss of the reversible neural network model to be trained is determined according to the first loss, the second loss, and the third loss.

[0076] As an example, the preset deformed content can be the image deformation removed in the process of converting the panorama image sample into the predicted deformed panorama image.

[0077] In a specific implementation, the first loss can be determined according to the panorama image sample and the super-resolution image of the predicted panorama image sample, for example, the first loss can be determined according to the similarity between the image contents, and the first loss can be negatively correlated with the similarity between the image contents, so that when the reversible neural network model is trained, the super-resolution image of the predicted panorama image sample output by the deformation restoration network can restore the deformation of the initial input panorama image sample as much as possible, and changes in the image content can be avoided.

[0078] And, a second loss can be obtained according to a correlation degree between the predicted deformed panorama image and the preset deformed content. Specifically, in a forward process of converting the panorama image sample into the predicted deformed panorama image, the panorama image sample has image deformation removed, and the removed image deformation is also referred to as preset deformed content or a hidden variable of the reversible neural network model. In a traditional manner, although the corresponding panorama image can be directly obtained based on the predicted deformed panorama image through network training, since the network training process does not record the lost information, that is, the preset deformed content, the original panorama image sample cannot be restored more accurately. In other words, panorama image sample = predicted deformed panorama image + preset deformed content. When the deformed content is not recorded, the restoration effect of the image will be affected. However, the preset deformed content is related to the input panorama image sample, that is, the preset deformed content of different panorama image samples is different.

[0079] Based on this, the preset deformed content can be obtained in the embodiment. The preset deformed content can be obtained by sampling from a statistical distribution of the deformed content. Then, the second loss is obtained according to the correlation degree between the predicted deformed panorama image and the preset deformed content. The second loss is negatively correlated with the correlation degree, so that the predicted deformed panorama image and the preset deformed content are as independent as possible.

[0080] Specifically, although the preset deformed content is related to the panorama image sample, the statistical distribution (for example, Gaussian distribution) of the preset deformed content can effectively assist in obtaining the information removed in the image deforming process. When the preset deformed content and the predicted deformed panorama image are independent of each other, and the deformed content is subject to a set statistical distribution, the preset deformed content in the deformation process can be safely discarded. When needed, the corresponding preset deformed content is obtained by sampling the statistical distribution.

[0081] Further, the third loss is obtained according to the deformed panorama image of the panorama image sample and the predicted deformed panorama image. For example, the third loss can be determined according to the similarity of the image content of the deformed panorama image of the panorama image sample and the predicted deformed panorama image. The third loss is negatively correlated with the similarity, so that the reversible neural network model can remove the image deformation in the panorama image sample while avoiding changes in other image information.

[0082] Further, the model loss of the reversible neural network model to be trained can be determined based on the determined first loss, second loss and third loss.

[0083] S250, the model parameters of the reversible neural network model to be trained are adjusted according to the model loss until a training end condition is met, and a trained reversible neural network model is obtained.

[0084] After the model loss is obtained, the model parameters of the reversible neural network model to be trained can be adjusted according to the model loss, and after the model parameters are adjusted this time, step S220 is returned to repeat the training of the reversible neural network model until the training end condition is met, for example, the iteration number of the reversible neural network model reaches the preset number or the model loss is within the preset range, and the trained reversible neural network model can be obtained. The trained reversible neural network model can be used in the image deformation removal stage to perform forward prediction using the deforming network to correct the image with content distortion into an un-distorted image, and after passing through the super-resolution network, the inverse prediction can be performed using the deformation restoration network to convert the un-distorted image back to the panoramic image with content distortion.

[0085] In this embodiment, by training the reversible neural network model, a basis can be provided for subsequent accurate removal of deformation content in the panoramic image and adaptation and restoration of the deformation content to the super-resolution image of the panoramic image, while fully preserving the image details.

[0086] In one embodiment, the method further comprises the following steps:

[0087] S410, obtaining a deformed panoramic image sample of a first resolution and a deformed panoramic image of a second resolution.

[0088] Wherein, the first resolution is less than the second resolution.

[0089] In actual application, the deformed panoramic image sample of the first resolution and the deformed panoramic image of the second resolution for training the super-resolution network can be obtained, wherein the deformed panoramic image sample of the first resolution can also be referred to as a low-resolution deformed panoramic image, and the deformed panoramic image of the second resolution can also be referred to as a high-resolution deformed panoramic image; the deformed panoramic image sample of the first resolution and the deformed panoramic image of the second resolution can have similar image content, and the difference between the two can only be the difference in image resolution.

[0090] S420, based on the deformed panoramic image sample of the first resolution and the deformed panoramic image of the second resolution, the discriminative network and the generative network are adversarially trained.

[0091] Further, the discriminative network and the generative network can be adversarially trained based on the deformed panoramic image sample of the first resolution and the deformed panoramic image of the second resolution.

[0092] In one embodiment, S420 can include the following steps:

[0093] The first resolution deformed panorama image sample is input into the generation network, the third resolution deformed panorama image is output by the generation network to the discrimination network, and the consistency discrimination result of the third resolution deformed panorama image and the second resolution deformed panorama image is output by the discrimination network; the network loss is determined according to the first resolution deformed panorama image sample, the second resolution deformed panorama image and the consistency discrimination result; and the discrimination network and the generation network are subjected to the adversarial training according to the network loss.

[0094] In a specific implementation, as shown in Figure 4 the first resolution deformed panorama image sample can be input into the generation network to be trained, and the generation network can refine the image details of the first resolution deformed panorama image sample to obtain the third resolution deformed panorama image, where the third resolution can be higher than the first resolution.

[0095] Further, the generation network can input the third resolution deformed panorama image into the discrimination network, and the second resolution deformed panorama image can be input into the discrimination network to trigger the discrimination network to output the consistency discrimination result of the third resolution deformed panorama image and the second resolution deformed panorama image. Specifically, the second resolution deformed panorama image is a reliable high-resolution image obtained in advance, and the third resolution deformed panorama image is obtained by refining the first resolution deformed panorama image sample and improving the image resolution. The discrimination network can determine whether the image resolution of the first resolution deformed panorama image sample reaches the second resolution after being processed by the generation network by judging whether the third resolution deformed panorama image is consistent with the second resolution deformed panorama image, and output the consistency discrimination result, where the consistency discrimination result can be a quantitative judgment, such as the consistency degree, or a qualitative judgment, such as consistency or inconsistency.

[0096] After obtaining the consistency discrimination result, the network loss can be determined in combination with the first resolution deformed panorama image sample, the second resolution deformed panorama image and the consistency discrimination result, and the discrimination network and the generation network are subjected to the adversarial training according to the network loss.

[0097] In an embodiment, the adversarial training of the discrimination network and the generation network according to the network loss can include:

[0098] The generation network and the discrimination network are alternately trained based on the network loss until a training end condition is met, and the trained generation network is obtained.

[0099] Specifically, the network loss can include a generation network loss and a discrimination network loss, the generation network loss can be determined based on the consistency discrimination result, when training the generation network and the discrimination network alternately based on the network loss, the network parameters of the discrimination network can be fixed first, the generation network is trained, the network parameters of the generation network are adjusted according to the generation network loss, gradient update is performed, when the training switching condition is met, the network parameters of the generation network are fixed, and the discrimination network is switched to be trained.

[0100] When training the discrimination network, the discrimination network loss can be determined according to the consistency discrimination result, that is, the discrimination network loss is obtained based on whether the discrimination network can correctly identify that the third resolution de-panoramic image output by the generation network is inconsistent with the second resolution de-panoramic image, and then the network parameters of the discrimination network are adjusted according to the discrimination network loss, gradient update is performed, when the training switching condition is met, the network parameters of the discrimination network are fixed, and the generation network is switched to be trained again, the generation network and the discrimination network are trained alternately for multiple times, until the training end condition is met, and the trained generation network and the trained discrimination network are obtained.

[0101] S430, the trained generation network is used as a super-resolution network.

[0102] After the generation network is trained, the trained generation network can be used as a super-resolution network.

[0103] In this embodiment, by obtaining the first resolution de-warping panoramic image sample and the second resolution de-warping panoramic image, the discrimination network and the generation network are trained in an adversarial manner based on the first resolution de-warping panoramic image sample and the second resolution de-warping panoramic image, the trained generation network can be used as a super-resolution network, and a basis is provided for subsequent super-resolution processing of images.

[0104] The embodiment of the application also provides a panoramic video super-resolution processing method, which can be applied to an application environment as shown in Figure 5 The application environment includes a terminal and a server, and the terminal communicates with the server through a network. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones or tablet computers and the like. The server can be implemented by an independent server or a server cluster composed of multiple servers.

[0105] In one embodiment, as shown in Figure 6 , a panoramic video super-resolution processing method is provided, which is taken as an example to illustrate the method applied to the server in Figure 5 , and includes the following steps:

[0106] S610, after receiving the panoramic video sent by the terminal, the panoramic video is framed to obtain multiple panoramic images.

[0107] In practical applications, the terminal can acquire panoramic video and send it to the server. For example, after capturing panoramic video using an application with panoramic video shooting capabilities, the terminal can directly upload it to the server; alternatively, it can capture panoramic video using dedicated panoramic video shooting equipment and then upload the panoramic video stored on the equipment to the server via the terminal.

[0108] After acquiring the panoramic video, it can be segmented into frames, and multiple panoramic images can be obtained based on the segmentation results. For example, video frames can be extracted from the panoramic video using video processing interfaces of computer vision and machine learning software libraries, such as using the API interface "cv2.VideoCapture()" in OpenCV.

[0109] S620, according to any of the above panoramic image super-resolution processing methods, obtain the super-resolution image of each frame of the panoramic image in the multi-frame panoramic image.

[0110] After acquiring multiple frames of panoramic images from the panoramic video, a super-resolution image of each frame of the panoramic image can be obtained according to the panoramic image super-resolution processing method described above.

[0111] Specifically, such as Figure 7 As shown, the panoramic video acquired by the terminal can also be called low-resolution panoramic video. After processing in step S610, multiple panoramic images, i.e., low-resolution panoramic images, can be obtained. Then, for each panoramic image in the multiple panoramic images, it can be input into a trained reversible neural network model. The deformation removal network in the reversible neural network model removes the image deformation in each panoramic image to obtain a deformed panoramic image, which can also be called an undistorted image. Then, the multiple deformed panoramic images can be input into a super-resolution network to obtain the super-resolution image of the deformed panoramic image, i.e., the undistorted high-resolution image. The multiple undistorted high-resolution images can be input into the reversible neural network model again. The deformation restoration network in this model restores the removed image deformation to the undistorted high-resolution image, resulting in the super-resolution image of each panoramic image output by the reversible neural network model, i.e., the high-resolution panoramic image.

[0112] S630: Based on the super-resolution image of each panoramic image in the multi-frame panoramic image, obtain the super-resolution panoramic video of the panoramic video.

[0113] The S640 sends the super-resolution panoramic video to the terminal.

[0114] After obtaining the super-resolution image of each frame of the panoramic image in the plurality of frames of panoramic images, the super-resolution panoramic video of the panoramic video can be generated in combination with the super-resolution images of the plurality of frames of panoramic images. For example, the super-resolution images of the plurality of frames of panoramic images can be combined into the super-resolution panoramic video by using a video processing interface of a computer vision and machine learning software library. For example, the API interface "cv2.VideoWriter()" of OpenCV can be used to realize the conversion from the low-resolution panoramic video to the high-resolution panoramic video. After processing, the super-resolution panoramic video of the panoramic video can be returned to the terminal.

[0115] In this embodiment, after receiving the panoramic video sent by the terminal, the panoramic video can be framed to obtain a plurality of frames of panoramic images. The super-resolution image of each frame of the panoramic image in the plurality of frames of panoramic images can be obtained according to any one of the panoramic image super-resolution processing methods described above. The super-resolution panoramic video of the panoramic video can be obtained according to the super-resolution images of each frame of the panoramic image in the plurality of frames of panoramic images. The super-resolution panoramic video of the panoramic video can be sent to the terminal. The scheme of this embodiment can make the panoramic video after super-resolution reconstruction retain more details, greatly improve the visual perception of the video, and effectively reduce the constraints and limitations of the video shooting hardware device on the video quality, and improve the user experience.

[0116] In addition, the scheme of this embodiment also provides an end-to-end super-resolution reconstruction framework for the video. The super-resolution panoramic video processed can be obtained by uploading the panoramic video to the server through the terminal, without further processing, thereby effectively improving the processing efficiency of the super-resolution panoramic video.

[0117] It should be understood that, although each step in the flowchart involved in each embodiment described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0118] In one embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 8. Figure 8As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is used to store panoramic images or panoramic videos. The network interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement the panoramic image super-resolution processing method or the panoramic video super-resolution processing method.

[0119] Those skilled in the art can understand that, Figure 8 The skilled in the art can understand that,

[0120] In one embodiment, a computer device is also provided, including a memory and a processor, the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0121] In one embodiment, a computer program product is provided, including a computer program, which is executed by a processor to implement the steps in the above method embodiments.

[0122] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties.

[0123] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read-Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0124] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictory, it should be considered as the scope of the present application.

[0125] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for super-resolution processing of a panoramic image, characterized in that, The method comprises: obtaining a panoramic image, and obtaining a pre-trained reversible neural network model; the reversible neural network model comprises a deformation removal network and a deformation restoration network corresponding to the deformation removal network, model parameters of the reversible neural network model are adjusted according to a model loss, the model loss is determined according to a panoramic image sample, a deformation-removed panoramic image corresponding to the panoramic image sample, a predicted deformation-removed panoramic image and a predicted super-resolution image of the panoramic image sample, the predicted deformation-removed panoramic image is obtained by processing the panoramic image sample through the deformation removal network in the reversible neural network model to be trained, the predicted super-resolution image of the panoramic image sample is obtained by processing a super-resolution image of the predicted deformation-removed panoramic image through the deformation restoration network in the reversible neural network model to be trained, and the super-resolution image of the predicted deformation-removed panoramic image is obtained by inputting the predicted deformation-removed panoramic image into a super-resolution network; inputting the panoramic image into the reversible neural network model, converting the panoramic image into a deformation-removed panoramic image through the deformation removal network in the reversible neural network model, and outputting the deformation-removed panoramic image to a super-resolution network, and obtaining a super-resolution image of the deformation-removed panoramic image through the super-resolution network; inputting the super-resolution image of the deformation-removed panoramic image into the deformation restoration network, and converting the super-resolution image of the deformation-removed panoramic image into a super-resolution image of the panoramic image through the deformation restoration network.

2. The method of claim 1, wherein, Before the panoramic image is obtained, the method further comprises: obtaining a panoramic image sample and a deformation-removed panoramic image of the panoramic image sample; inputting the panoramic image sample into the reversible neural network model to be trained, outputting a predicted deformation-removed panoramic image to the super-resolution network through the deformation removal network in the reversible neural network model to be trained, so that the super-resolution network outputs a super-resolution image of the predicted deformation-removed panoramic image; inputting the super-resolution image of the predicted deformation-removed panoramic image into the reversible neural network model to be trained, and outputting a predicted super-resolution image of the panoramic image sample through the deformation restoration network in the reversible neural network model to be trained; determining a model loss of the reversible neural network model to be trained according to the panoramic image sample, the deformation-removed panoramic image of the panoramic image sample, the predicted deformation-removed panoramic image and the predicted super-resolution image of the panoramic image sample; adjusting model parameters of the reversible neural network model to be trained according to the model loss until a training end condition is met, and obtaining a trained reversible neural network model.

3. The method of claim 2, wherein, The method of determining the model loss of the reversible neural network model to be trained according to the panoramic image sample, the deformation-removed panoramic image of the panoramic image sample, the predicted deformation-removed panoramic image and the predicted super-resolution image of the panoramic image sample comprises: a first loss is obtained according to the panorama image sample and a super-resolution image of the predicted panorama image sample; a second loss is obtained according to a correlation degree between the predicted deformed panorama image and preset deformed content; a third loss is obtained according to the deformed panorama image of the panorama image sample and the predicted deformed panorama image; a model loss of the reversible neural network model to be trained is determined according to the first loss, the second loss and the third loss.

4. The method of claim 2, wherein, The method further comprises: The deformed panorama image of the panorama image sample is obtained by converting the panorama image sample in a spherical image projection form into an image in a cubic image projection form.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: a deformed panorama image sample of a first resolution and a deformed panorama image of a second resolution are obtained; the first resolution is less than the second resolution; adversarial training is performed on the discriminative network and the generative network based on the deformed panorama image sample of the first resolution and the deformed panorama image of the second resolution; the trained generative network is used as a super-resolution network.

6. The method of claim 5, wherein, The adversarial training performed on the discriminative network and the generative network based on the deformed panorama image sample of the first resolution and the deformed panorama image of the second resolution comprises: the deformed panorama image sample of the first resolution is input into the generative network, and a deformed panorama image of a third resolution is output by the generative network to the discriminative network, so that the discriminative network outputs a consistency discrimination result of the deformed panorama image of the third resolution and the deformed panorama image of the second resolution; a network loss is determined according to the deformed panorama image sample of the first resolution, the deformed panorama image of the second resolution and the consistency discrimination result; adversarial training is performed on the discriminative network and the generative network based on the network loss.

7. The method of claim 6, wherein, The adversarial training performed on the discriminative network and the generative network based on the network loss comprises: the generative network and the discriminative network are alternately trained based on the network loss until a training end condition is met, and a trained generative network is obtained.

8. A panoramic video super-resolution processing method, characterized in that, The method comprises: after receiving a panorama video sent by a terminal, the panorama video is framed to obtain a plurality of panorama images; the super-resolution image of each of the plurality of panorama images is obtained according to the method of any one of claims 1 to 7; a super-resolution panorama video of the panorama video is obtained according to the super-resolution image of each of the plurality of panorama images; the super-resolution panorama video of the panorama video is sent to the terminal. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 8.

10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 8.

Images