Panoramic image super-resolution method and system, electronic device, and storage medium

Abstract

Disclosed are a panoramic image super-resolution method and system, an electronic device, and a storage medium. The method comprises: establishing a panoramic image super-resolution model based on latitude-graded enhancement, the model comprising a feature extraction sub-network, a deep feature enhancement sub-network, an upsampling sub-network, and a reconstruction sub-network, wherein the feature extraction sub-network is used for extracting shallow features of an input panoramic image; the deep feature enhancement sub-network, by means of cascaded latitude-graded enhancement units, performs graded deep enhancement on shallow features of different latitude regions, and refines high-frequency information of different latitudes on the basis of differences in feature distribution across different latitude regions in the panoramic image, so as to obtain enhanced features; the upsampling sub-network uses sub-pixel convolution layers to amplify the spatial resolution dimensions of the enhanced features; and the reconstruction sub-network restores the amplified enhanced features into a high-resolution panoramic image. The present invention solves the problem of differences in feature distribution across different latitudes in panoramic images, and improves the super-resolution performance of panoramic images in various scene environments.

Description

Panoramic image super-resolution method, system, electronic device and storage medium TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and specifically relates to a panoramic image super-resolution method, system, electronic device and storage medium. BACKGROUND

[0002] With the development and popularization of panoramic image technology, people's demand for high-resolution panoramic images is growing. Panoramic images can capture the complete view of a scene, which has wide applications in virtual reality, urban planning, architectural design and tourism. High-resolution panoramic images can provide a 360-degree view, which means that every detail in the image can be clearly displayed, and this is crucial for enhancing user experience and improving the practical utility of applications. For example, in virtual reality, high-resolution panoramic images can provide users with a more immersive experience, making them feel as if they are in a real environment; in urban planning, high-resolution panoramic images can help planners more accurately analyze the distribution of terrain and buildings; in tourism, high-resolution panoramic images can allow potential tourists to more intuitively understand the 360-degree scenery of the destination. However, compared with high-resolution ordinary pictures and videos, the shooting of high-resolution panoramic images not only requires high-precision optical equipment and image sensors for shooting, but also requires the combination and registration of pictures taken from different angles to construct the final 360-degree panoramic image, which results in higher shooting costs. Especially in outdoor or complex scenes, the difficulty and cost of obtaining high-quality panoramic images will further increase.

[0003] Therefore, in the case that existing devices cannot meet the demand for high-resolution panoramic images, the resolution of most panoramic images is relatively low, while users' demand for high-definition panoramic images is constantly rising. This contradiction in demand has driven the development of panoramic image super-resolution technology, which can convert low-resolution panoramic images into high-resolution images without replacing hardware devices, thus meeting the actual needs of various applications.

[0004] However, panoramic images are usually stored in equirectangular projection (ERP) format, which has serious geometric distortion and latitude-related feature distribution differences. In the ERP panorama, starting from the equator, as the latitude rises towards the south or north pole, the texture features and detailed information in the panorama significantly decrease. From a mathematical point of view, ERP projection is a process of projecting from a sphere to a plane, which has serious spatial distortion. Since this stretching (also known as oversampling) is usually performed by interpolation operations (such as bicubic interpolation), it will inevitably cause the texture and detail information at high latitudes to become smoother. Therefore, the ERP panorama presents obvious texture and detail distribution differences. The existing image super-resolution methods, most of which are designed for ordinary 2D images, cannot effectively solve this problem, and it is difficult to obtain ideal super-resolution performance, which cannot meet the user's visual needs. SUMMARY

[0005] The present application is directed to the above problems, and provides a panoramic image super-resolution method, system and storage medium based on latitude hierarchical reinforcement, aiming to restore a higher resolution and higher quality panoramic image.

[0006] According to a first aspect of the embodiments of the present disclosure, a panoramic image super-resolution method based on latitude hierarchical reinforcement is provided, and the method comprises the following steps:

[0007] Obtaining a panoramic image dataset containing multiple different scenes, and dividing the panoramic image dataset into a training dataset and a test dataset after preprocessing the panoramic image dataset;

[0008] Establishing a panoramic image super-resolution model based on latitude hierarchical reinforcement, wherein the panoramic image super-resolution model comprises a feature extraction subnetwork, a deep feature enhancement subnetwork, an upsampling subnetwork, and a reconstruction subnetwork. The feature extraction subnetwork is used to extract shallow features of an input panoramic image. The deep feature enhancement subnetwork enhances the shallow features of different latitude regions in a hierarchical manner through a cascaded latitude hierarchical reinforcement unit, extracts high-frequency information of different latitudes based on the difference in feature distribution of different latitude regions in the panoramic image, and obtains enhanced features. The upsampling subnetwork uses a sub-pixel convolution layer to enlarge the spatial resolution size of the enhanced features. The reconstruction subnetwork restores the enlarged enhanced features into a high-resolution panoramic image.

[0009] Training and testing the panoramic image super-resolution model using the training dataset and the test dataset, respectively,

[0010] Implementing panoramic image super-resolution using the panoramic image super-resolution model that has passed training and testing.

[0011] In some embodiments, the training and testing of the panoramic image super-resolution model using the training data set and the test data set specifically include:

[0012] The training data set under multiple different scenarios is sequentially input into the panoramic image super-resolution model;

[0013] When iteratively training, the loss value between the high-resolution panoramic image generated by the panoramic image super-resolution model and the true value is calculated;

[0014] The panoramic image super-resolution model is trained with the optimization goal of reducing the loss value, and the network weight is updated and optimized synchronously;

[0015] The test data set is used to evaluate the reconstruction performance of the panoramic image super-resolution model.

[0016] In some embodiments, the deep feature enhancement subnetwork divides the shallow features into different regions according to the latitude, and performs hierarchical enhancement on the features in different regions, that is, the strength of feature extraction increases from high-latitude regions to low-latitude regions.

[0017] In some embodiments, the hierarchical reinforcement unit further combines the enhanced features in each region into a feature map, and enhances the feature map using a cascaded convolution layer.

[0018] In some embodiments, the loss value is obtained by a global loss function, and the specific expression of the global loss function is:

[0019] ,

[0020] wherein, is the number of samples in the training set, , and respectively represent the super-resolution result of the panoramic image super-resolution model for the kth input panoramic image and its corresponding true value, is the global loss function.

[0021] According to a second aspect of the embodiments of the present disclosure, a panoramic image super-resolution system based on latitude hierarchical reinforcement is provided, and the system comprises:

[0022] A data set acquisition module is configured to acquire a panoramic image data set containing multiple different scenarios, and divide the panoramic image data set into a training data set and a test data set after preprocessing the panoramic image data set;

[0023] The model establishing module is configured to establish a panoramic image super-resolution model based on latitude hierarchical reinforcement, which comprises a feature extraction subnetwork, a deep feature enhancement subnetwork, an upsampling subnetwork, and a reconstruction subnetwork. The feature extraction subnetwork is configured to extract shallow features of an input panoramic image. The deep feature enhancement subnetwork is configured to perform hierarchical deep enhancement on the shallow features of different latitude regions through a cascade of latitude hierarchical reinforcement units, extract high-frequency information of different latitudes based on the difference in feature distribution of different latitude regions in the panoramic image, and obtain enhanced features. The upsampling subnetwork is configured to enlarge the spatial resolution size of the enhanced features by using a sub-pixel convolution layer. The reconstruction subnetwork is configured to restore the enlarged enhanced features into a high-resolution panoramic image.

[0024] The model training and testing module is configured to train and test the panoramic image super-resolution model by using the training data set and the test data set, respectively.

[0025] The panoramic image super-resolution implementation module is configured to implement panoramic image super-resolution by using the panoramic image super-resolution model that has passed the training and the test.

[0026] In some embodiments, the model training and testing module comprises the following implementation process:

[0027] The training data set in multiple different scenes is sequentially input into the panoramic image super-resolution model.

[0028] During iterative training, the loss value between the high-resolution panoramic image generated by the panoramic image super-resolution model and the true value is calculated.

[0029] The panoramic image super-resolution model is trained with the optimization goal of reducing the loss value, and the network weight is updated and optimized synchronously.

[0030] The test data set is used to evaluate the reconstruction performance of the panoramic image super-resolution model.

[0031] In some embodiments, the deep feature enhancement subnetwork divides the shallow features into different regions according to the latitude height, and performs hierarchical enhancement on the features in different regions, that is, the strength of feature extraction obtained from high latitude regions to low latitude regions increases in turn.

[0032] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the panoramic image super-resolution method based on latitude hierarchical reinforcement are implemented.

[0033] According to a fourth aspect of the embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, and the storage medium has stored thereon computer instructions, which, when executed by a processor, implement the steps of the panoramic image super-resolution method based on latitude hierarchical enhancement.

[0034] The panoramic image super-resolution method, system and storage medium based on latitude hierarchical enhancement provided by the embodiments of the present disclosure have the following beneficial effects:

[0035] The embodiments of the present disclosure fully consider the difference in feature distribution of different latitude regions in the panoramic image, divide the features of the panoramic image into different regions according to the latitude, enhance the features of these different regions in different levels through the cascaded latitude hierarchical enhancement unit, ensure that the different latitude regions are enhanced to different degrees, that is, the lower latitude region with more texture features will obtain stronger refinement. Through this unique differential refinement of deep features, the high-frequency information of different latitudes can be fully extracted, so that clearer details, textures and object edges are restored. In addition, in order to solve the information isolation of different regions caused by the spatial division, after the features of each region are merged into a complete feature map in the hierarchical enhancement unit, the cascaded convolution layer is further used to enhance the complete feature map, promote the information interaction between different regions, avoid the occurrence of edge artifacts, and thus improve the reconstruction performance of super-resolution.

[0036] In summary, the present application effectively solves the problem of the difference in feature distribution of different latitudes in the panoramic image, improves the super-resolution effect of the panoramic image in different scene environments, and enhances the quality and visual experience of the panoramic image.

[0037] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application,

[0039] FIG. 1 is a flowchart of a panoramic image super-resolution method based on latitude hierarchical enhancement according to an embodiment of the present application;

[0040] FIG. 2 is a schematic diagram of the network structure of a panoramic image super-resolution model according to an embodiment of the present application;

[0041] FIG. 3 is a flowchart of a hierarchical enhancement unit in a deep feature enhancement subnetwork according to an embodiment of the present application;

[0042] FIG. 4 is a schematic diagram of the structure of a panoramic image super-resolution system based on latitude hierarchical enhancement according to an embodiment of the present application;

[0043] Fig. 5 is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0044] The application will be further described below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the application, but not to limit the application. In addition, it should be noted that, for the convenience of description, only the parts related to the application are shown in the drawings, but not all the structures.

[0045] Before the example embodiments are discussed in more detail, it should be mentioned that some of the example embodiments are described as processes or methods depicted as flow charts. While the steps of the processes are depicted in a sequential order, many of the steps can be performed in parallel, concurrently or at the same time. In addition, the order of the steps can be re-arranged. The processes can be terminated when their operations are completed, but can also have additional steps not included in the figure. The processes can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0046] Embodiments of the present application are directed to a panorama image super-resolution method based on latitude hierarchical enhancement, a system and a storage medium, and provide the following embodiments:

[0047] In the specific implementation process, the super-resolution of the panorama image is different from the super-resolution of the ordinary natural picture. The panorama image usually adopts an equirectangular projection (ERP) format, and there is a serious geometric distortion and a difference in the texture features related to the latitude. In the ERP panorama, the high-latitude area (north latitude), the spatial distortion is large, and the texture features are less; the area near the equator, the spatial distortion is small, and the texture features are more; the high-latitude area (south latitude), the spatial distortion is large, and the texture features are less. That is, as the latitude continuously rises towards the south pole or the north pole, the texture features and the detail information in the panorama image will obviously become less. From the mathematical point of view, the ERP projection is a process of projecting from a spherical surface to a plane, and suffers from serious spatial distortion and oversampling. For example, the pixels of the poles on the spherical surface can even be stretched into a line in the ERP plane, and such stretching and oversampling are usually performed by interpolation operations (such as bicubic interpolation and linear interpolation), which inevitably leads to the high-frequency information in the high-latitude area becoming smoother, so that the texture features become less. Therefore, when performing the super-resolution of the panorama image, it is necessary to adapt to the spatial distortion of the panorama image and consider the difference in the distribution of the texture features related to the latitude. However, the existing image super-resolution technology is mostly for ordinary natural images, and cannot realize high-performance super-resolution of the panorama image.

[0048] Based on these constraints, the key problem to be solved by the panoramic image super-resolution method is to realize high-performance super-resolution of input panoramic images in different environments and efficiently extract and enhance high-frequency information of different latitudes.

[0049] The first embodiment is used to illustrate the panoramic image super-resolution method based on latitude hierarchical reinforcement, and refers to FIG. 1, which is a flow chart of the panoramic image super-resolution method based on latitude hierarchical reinforcement. The specific steps include:

[0050] S1, acquiring a panoramic image dataset containing multiple different scenes, and dividing the panoramic image dataset into a training dataset and a test dataset after preprocessing the panoramic image dataset;

[0051] In the specific implementation process, panoramic images can be collected by different methods to construct a dataset for model training and testing. For example, panoramic pictures can be downloaded from a public free database; keywords of different scene environments can be searched through a search engine platform, and corresponding free panoramic pictures can be downloaded; and a mobile phone, a camera or other shooting devices can be used to capture the required panoramic pictures.

[0052] In the specific implementation process, the panoramic picture dataset at least includes various panoramic pictures of more than 20 environments and scenes, and the dataset is cleaned through preprocessing operations such as format unification, resolution unification and deletion of abnormal pictures. Then, the dataset is randomly divided into a training dataset (training set) and a test dataset (test set). The training set accounts for 90%, which is used to train the super-resolution model of the present application, and the test set accounts for 10%, which is used to evaluate the performance of the model. The low-resolution panoramic image is input into the super-resolution model of the present application, and the corresponding high-resolution panoramic image can be generated.

[0053] S2, establishing a panoramic image super-resolution model based on latitude hierarchical reinforcement, wherein the panoramic image super-resolution model includes a feature extraction subnetwork, a deep feature enhancement subnetwork, an upsampling subnetwork and a reconstruction subnetwork. The feature extraction subnetwork is used to extract shallow features of an input panoramic picture. The deep feature enhancement subnetwork enhances the shallow features of different latitude regions in different levels through a cascaded latitude hierarchical reinforcement unit, extracts high-frequency information of different latitudes based on the difference in feature distribution of different latitude regions in the panoramic image, and obtains enhanced features. The upsampling subnetwork uses a sub-pixel convolution layer to enlarge the spatial resolution size of the enhanced features. The reconstruction subnetwork restores the enlarged enhanced features into high-resolution panoramic images.

[0054] S3, training and testing the panoramic image super-resolution model using the training dataset and the test dataset, respectively;

[0055] S4, implementing panoramic image super-resolution by using the panoramic image super-resolution model passed by training and testing.

[0056] Training and testing the panoramic image super-resolution model by using the training data set and the testing data set, specifically comprising:

[0057] Inputting the training data set in different scenes into the panoramic image super-resolution model in sequence;

[0058] When iteratively training, calculating the loss value between the high-resolution panoramic image generated by the panoramic image super-resolution model and the true value;

[0059] Training the panoramic image super-resolution model with the optimization target of reducing the loss value, and synchronously updating and optimizing the network weight;

[0060] Evaluating the reconstruction performance of the panoramic image super-resolution model by using the testing data set.

[0061] The deep feature enhancement subnetwork divides the shallow features into different regions according to the latitude, and performs hierarchical deep enhancement on the shallow features in different regions, that is, the strength of feature extraction increases in turn from the high-latitude region to the low-latitude region.

[0062] The hierarchical enhancement unit also merges the enhanced features in each region into a feature map, and enhances the feature map by using a cascaded convolution layer.

[0063] In the implementation process, the pre-established panoramic image super-resolution model based on latitude hierarchical enhancement is initialized, and various training and testing hyperparameters required by the neural network model are set, such as the feature channel number of the model, the network layer number, the learning rate, the optimizer, the iteration number, etc.

[0064] The super-resolution model of the panoramic image mainly includes a feature extraction subnetwork, a feature enhancement subnetwork, an upsampling subnetwork, and a reconstruction subnetwork. As shown in FIG. 2, in the super-resolution model of the present application, the feature extraction subnetwork uses a certain number of convolution layers to extract the shallow features of the input panoramic image in different environments; the deep feature enhancement subnetwork subdivides the feature map of the panoramic image into multiple different sub-regions according to the latitude, independently and hierarchically enhances the features in these sub-regions, so that the low-latitude region with more texture features will obtain stronger extraction. Through this differential enhancement of deep features, the high-frequency information of each latitude can be fully extracted and saved, so that clearer details, textures and edge structures can be restored in the generated high-resolution panoramic image; the upsampling subnetwork uses a deep learning-based upsampling operation to enlarge the resolution of the enhanced feature map; and the reconstruction subnetwork restores the enlarged feature into a high-resolution panoramic image, and then outputs it.

[0065] The feature extraction subnetwork can effectively extract the features of the input panoramic image through one layer or a plurality of convolution layers in cascade, facilitating the deep feature enhancement subnetwork to implement further deep and efficient enhancement.

[0066] The deep feature enhancement subnetwork fully considers the difference in texture feature distribution related to latitude in the panoramic image, effectively explores and refines high-frequency information at different latitudes, and thus realizes efficient detail and texture recovery, thereby obtaining better panoramic image super-resolution effect. Specifically, the deep feature enhancement subnetwork is composed of N cascaded hierarchical reinforcement units, through which the differential reinforcement of deep features can be realized. Like the number of channels of the network, the number N of hierarchical reinforcement units can be set as a hyperparameter in the training process according to actual application requirements, such as 5, 8, 10, etc. The input of the first hierarchical reinforcement unit is the shallow feature extracted by the feature extraction subnetwork, denoted as , and the output feature of the nth hierarchical reinforcement unit is denoted as . The deep enhancement of the features by the N cascaded hierarchical reinforcement units can be expressed as follows:

[0067]

[0068] wherein represents the function of the nth hierarchical reinforcement unit, . Note that the input of the nth hierarchical reinforcement unit is the output of the (n-1)th hierarchical reinforcement unit.

[0069] Figure 3 clearly shows the network structure and data flow of each hierarchical reinforcement unit. The hierarchical reinforcement unit first implements graded reinforcement of the features of the panoramic image based on the high and low latitudes. Specifically, there are J reinforcement levels in each hierarchical reinforcement unit, and J is also a hyperparameter preset in the training process. For ease of illustration, taking the nth hierarchical reinforcement unit as an example, the input feature map thereof is . First, it can be divided into 2J sub-regions of the same size along the latitude, i.e. , note that the feature map of the jth sub-region can be denoted as , and h=H / 2L is the height of each sub-region, wherein H represents the height of the feature map .

[0070] Then, different levels of reinforcement are implemented on these sub-regions (which can also be referred to as sub-region feature maps). Considering that the spatial distortion and stretching in the panoramic image are symmetrical between the southern and northern hemispheres, a single level of reinforcement needs to process a pair of geometrically symmetrical sub-regions. Specifically, in the jth level of reinforcement, the jth sub-region and the (2L+1−j)th sub-region are simultaneously enhanced​ As shown in FIG. 3, in the j-th level of enhancement, j convolutional layers are stacked together. Therefore, different levels of enhancement have different depths of convolutional networks, so as to realize hierarchical and differential enhancement of the feature map of the entire panoramic image. That is, as the latitude of the sub-region decreases, the more layers of network used by the hierarchical enhancement unit, the higher the enhancement level, and the more obvious the enhancement of the low-latitude area with complex texture and details. This improves the adaptability to spatial distortion and oversampling in the panoramic image, and fully considers the difference in the distribution of texture features related to latitude in the panoramic image. This is because, compared with high-latitude areas, low-latitude areas have more texture and details, and it is more difficult to perform super-resolution reconstruction on them, so a deeper neural network is needed to enhance them. The above process of hierarchical enhancement can be described by the following formula:

[0071]

[0072]

[0073]

[0074] wherein, is the operation of segmenting the features of the panoramic image, is a function of the convolutional layer in the j-th level of enhancement, and are the sub-regions and are the output results of the hierarchical enhancement.

[0075] After enhancing these sub-regions, they need to be merged again to generate a complete feature map. Assuming denotes the merging operation, then this process is represented as follows:

[0076] wherein, is the feature map after merging. In addition, the residual technique is also used inside the hierarchical enhancement unit, that is, the merged feature map is fused with the input by element-wise addition. The application of the residual technique can improve the training efficiency and representation ability, and can avoid gradient explosion in the training process.

[0077] ​​​​It should be noted that in this embodiment, the j-th level of reinforcement can be composed of j ordinary convolutions by default. In order to save the cost of calculation, it can also be set as j group convolutions to build a more efficient network model. Compared with ordinary convolution, group convolution can greatly reduce the amount of calculation while maintaining satisfactory performance. Secondly, the bottleneck structure can also be designed, and a 1x1 convolution is used before the j convolution layers to compress the channel number to 1 / r of the original, and finally a 1x1 convolution is used to restore the original channel number.

[0078] Because this spatial division will cause the information between different sub-regions to be isolated, affecting the modeling effect, and even producing edge artifacts. In order to solve this problem, after merging the features of each region into a complete feature map in the hierarchical reinforcement unit, one or more layers of cascaded convolution can be used to further enhance the complete feature map, which can promote the interaction of information between different regions and avoid the appearance of edge artifacts, thereby enhancing the reconstruction performance of super-resolution. This process can be represented by the following formula:

[0079]

[0080] is a function of the one or more layers of cascaded convolution. After the re-modeling of the convolution layer, the context information is modeled with a longer distance dependency, thereby promoting the performance improvement of the super-resolution model, and obtaining the output of the n-th hierarchical reinforcement unit .

[0081] The depth enhancement subnetwork designs a hierarchical reinforcement scheme, which fully adapts to the spatial distortion of the panoramic image and the difference in feature distribution caused by it, and realizes the extraction of high-frequency information at different latitudes in a spatial refinement manner, effectively models the depth features of the panoramic image, and thus improves the reconstruction performance of the panoramic image.

[0082] The upsampling subnetwork is used to enlarge the resolution of the enhanced features. That is, the size of the feature map is expanded to a specified multiple based on deep learning technology (sub-pixel convolution layer), and the object texture and details and other information are not damaged. The sub-pixel convolution layer maps the number of channels to a specified multiple, and then rearranges the features at the same spatial position in multiple channels in the same channel, thereby realizing the enlargement of the spatial resolution of the feature map.

[0083] ​The reconstruction subnetwork is used for reconstructing the enlarged size feature map output by the upsampling subnetwork into a high-definition and high-resolution panoramic image (RGB, three channels) through 1 layer of convolution, and finally realizes the panoramic image super-resolution method based on latitude hierarchical reinforcement, so that the method has good super-resolution effect on panoramic image data in different scenes and different environments. Further, the panoramic image super-resolution neural network model based on latitude hierarchical reinforcement is successfully established.

[0084] It should be clarified that, in this embodiment, the feature extraction subnetwork, the upsampling subnetwork and the reconstruction subnetwork are not described in detail using corresponding formulas for the sake of clear and concise expression. However, the technology and details used therein have been clearly explained in the present application, so that a person skilled in the art should be able to recognize the specific functions of the subnetworks.

[0085] Different panoramic images are selected from the training set and input into the panoramic image super-resolution neural network model based on latitude hierarchical reinforcement in turn, and continuous training is performed. The input panoramic images should include 20 scenes and environments or more, so that the adaptability of the model in different environments is significantly enhanced, thereby improving the practicality of the super-resolution method.

[0086] Different panoramic images are selected from the training set in turn, and the following steps are iteratively executed: panoramic images of different scenes in the training set are input into the model in turn, the model is iteratively trained, the loss value is converged, and the model under complex motion scenes is generated. In each iteration training, a certain number of panoramic images are selected, the selected panoramic images include low-resolution panoramic images and corresponding high-resolution images as true values for supervised training. The true value image and the output image of the model are input into the pre-set loss function to calculate the global loss value, and the network weight of the super-resolution model is continuously updated by training. During the iteration process, the necessity of re-parameterization and training can be judged according to the learning rate change and the loss value decrease law, and the process is repeated. After the training is completed, the loss value should be converged, and the final model can be generated.

[0087] The global loss value of the panoramic image super-resolution network is calculated. The panoramic image super-resolution network selects multiple batches of panoramic images from the data set at each iteration training, inputs them into the model, and calculates the loss value based on the loss function pre-set for the panoramic image super-resolution network.

[0088] The panoramic image super-resolution model based on latitude hierarchical reinforcement adopts L1 Loss function as the loss function. Compared with L2 loss, the L1 loss function has better robustness, so that the restored panoramic image has more real texture and details. Moreover, the L1 loss function has better tolerance to abnormal values, which can improve the robustness of the model.

[0089] In order to globally optimize the super-resolution network model, the present application needs to input the panoramic image to be super-resolved into the super-resolution model based on latitude hierarchical reinforcement, and after obtaining the predicted high-resolution panoramic image, input the predicted panoramic image and the corresponding true value panoramic image into the L1 loss function, and calculate the final global loss value.

[0090] In the optional implementation of the present embodiment, the loss value of the super-resolution model based on latitude hierarchical reinforcement, i.e. the final global loss value, is calculated. It can be assumed that are the super-resolution results of the model for the kth input panoramic image and the corresponding true value, respectively, K is the number of samples (i.e. the number of panoramic images) in the training set, is the global loss function. Then, the L1 loss function calculation formula mentioned in the present application is as follows:

[0091]

[0092] According to the pre-set learning rate, the present application sets the learning rate to 2e-4, then uses the Adaw optimizer for iterative training, and updates the weights of each subnet in the network, so that the loss value continuously decreases. When the loss value converges, the corresponding model parameters can be saved to construct the panoramic image super-resolution model based on latitude hierarchical reinforcement.

[0093] According to the pre-divided data set, the panoramic images in the test set are selected to test the generated panoramic image super-resolution model based on latitude hierarchical reinforcement, so as to verify the reconstruction performance of the model. After successful testing, it can be used as the final panoramic image super-resolution model based on latitude hierarchical reinforcement.

[0094] Another embodiment is used to illustrate the panoramic image super-resolution system based on latitude hierarchical reinforcement. Referring to FIG. 4, the system 400 includes:

[0095] The data set acquisition module 410 is used to acquire a panoramic image data set containing multiple different scenes, and divide the panoramic image data set into a training data set and a test data set after preprocessing the panoramic image data set;

[0096] ​​The model establishing module 420 is configured to establish a panoramic image super-resolution model based on latitude hierarchical reinforcement, wherein the panoramic image super-resolution model comprises a feature extraction subnetwork, a deep feature enhancement subnetwork, an upsampling subnetwork, and a reconstruction subnetwork. The feature extraction subnetwork is configured to extract shallow features of an input panoramic image. The deep feature enhancement subnetwork is configured to perform hierarchical deep enhancement on the shallow features of different latitude regions through a cascade of latitude hierarchical reinforcement units, extract high-frequency information of different latitudes based on the difference in feature distribution of different latitude regions in the panoramic image, and obtain enhanced features. The upsampling subnetwork is configured to enlarge the spatial resolution size of the enhanced features by using a sub-pixel convolution layer. The reconstruction subnetwork is configured to restore the enlarged enhanced features into a high-resolution panoramic image.

[0097] The model training and testing module 430 is configured to train and test the panoramic image super-resolution model by using the training data set and the test data set, respectively.

[0098] The panoramic image super-resolution implementation module 440 is configured to implement panoramic image super-resolution by using the panoramic image super-resolution model that passes the training and testing.

[0099] The implementation process of the model training and testing module 430 specifically comprises the following steps.

[0100] The training data set in multiple different scenes is sequentially input into the panoramic image super-resolution model.

[0101] When iteratively training, a loss value between a high-resolution panoramic image generated by the panoramic image super-resolution model and a true value is calculated.

[0102] The panoramic image super-resolution model is trained with the optimization goal of reducing the loss value, and the network weight is updated and optimized synchronously.

[0103] The test data set is used to evaluate the reconstruction performance of the panoramic image super-resolution model.

[0104] The deep feature enhancement subnetwork divides the shallow features into different regions according to the latitude height, and performs hierarchical enhancement on the features in different regions, that is, the strength of feature extraction obtained from the high-latitude region to the low-latitude region is sequentially increased.

[0105] In addition to the above modules, the system 400 can also include other components. However, since these components are irrelevant to the content of the embodiments of the present disclosure, the illustrations and descriptions thereof are omitted here.

[0106] For other specific working processes of the panoramic image super-resolution system 400 based on latitude hierarchical reinforcement, refer to the description of the panoramic image super-resolution method based on latitude hierarchical reinforcement described above, and no further description is given here.

[0107] The panoramic image super-resolution system based on latitude hierarchical enhancement provided in the embodiment can enhance the input panoramic image across latitude through the hierarchical enhancement mechanism, adapt to the distortion of panoramic image, and has efficient information extraction capability, and has significant compatibility for panoramic images of different scenes. Therefore, the super-resolution device can realize higher panoramic image super-resolution performance at lower calculation cost, and can be well popularized and applied in different fields (such as virtual reality and 3D vision).

[0108] Another embodiment is used to illustrate that the system of the present application can also be implemented by means of the architecture of the computing device shown in FIG. 5. FIG. 5 shows the architecture of the computing device. As shown in FIG. 5, the computer system 510, the system bus 530, the one or more CPUs 540, the input / output 520, the memory 550, etc. The memory 550 can store various data or files used by the computer processing and / or communication and the program instructions executed by the CPU including the program instructions of the embodiment of the method of the panoramic image super-resolution based on latitude hierarchical enhancement. The architecture shown in FIG. 5 is only exemplary, and when implementing different devices, one or more components in FIG. 5 are adjusted according to actual needs. The memory 550 as a kind of computer readable storage medium can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the method of the panoramic image super-resolution based on latitude hierarchical enhancement in the embodiment of the present application (for example, the data set acquisition module 410, the model establishment module 420, the model training and testing module 430 and the panoramic image super-resolution implementation module 440 in the panoramic image super-resolution system 400 based on latitude hierarchical enhancement). The one or more CPUs 540 execute various functions of the system of the present application and data processing by running the software programs, instructions and modules stored in the memory 550, that is, implement the above-mentioned method of the panoramic image super-resolution based on latitude hierarchical enhancement, which includes the following steps:

[0109] obtaining a panoramic image data set containing a plurality of different scenes, and dividing the panoramic image data set into a training data set and a test data set after preprocessing the panoramic image data set;

[0110] A panorama image super-resolution model based on latitude hierarchical reinforcement is established, wherein the panorama image super-resolution model comprises a feature extraction subnetwork, a deep feature enhancement subnetwork, an upsampling subnetwork and a reconstruction subnetwork, wherein the feature extraction subnetwork is used to extract shallow features of an input panorama picture; the deep feature enhancement subnetwork enhances the shallow features of different latitude regions in a hierarchical manner through a cascaded latitude hierarchical reinforcement unit, extracts high-frequency information of different latitudes based on the difference in feature distribution of different latitude regions in the panorama picture, and obtains enhanced features; the upsampling subnetwork uses a subpixel convolution layer to enlarge the spatial resolution size of the enhanced features; and the reconstruction subnetwork restores the enlarged enhanced features into a high-resolution panorama picture.

[0111] The panorama image super-resolution model is trained and tested by using the training data set and the test data set respectively,

[0112] The panorama image super-resolution model that passes the training and the test is used to realize panorama image super-resolution.

[0113] Of course, the processor of the server provided in the embodiments of the present application is not limited to performing the method operations as described above, but can also perform related operations in the panorama image super-resolution method based on latitude hierarchical reinforcement provided in any embodiment of the present application.

[0114] The memory 550 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required by a function; and the data storage area can store data created according to the use of the terminal and the like. In addition, the memory 550 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some examples, the memory 550 can further include a memory remotely arranged with respect to the one or more CPUs 540, and the remote memory can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0115] The input / output 520 can be used to receive input digital or character information, and to generate key signal inputs related to the user settings and function control of the device. The input / output 520 can also include a display device such as a display screen.

[0116] The embodiment of the present application also provides a non-temporary computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the panoramic image super-resolution method based on latitude hierarchical reinforcement described in the above embodiment. The computer readable storage medium of the embodiment of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, device or component.

[0117] The computer readable signal medium can include a data signal propagated in a baseband or propagated as a carrier wave in a propagated data signal, in which the computer readable program code is contained. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate or transmit the program for use by or in connection with an instruction execution system, device or component.

[0118] The program code contained on the storage medium can be transmitted in any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0119] In addition, the other specific working process of the non-temporary computer readable storage medium is described with reference to the above embodiment of the panoramic image super-resolution method based on latitude hierarchical reinforcement, and will not be described again.

[0120] According to the technical solutions provided in the above embodiments, the panorama image super-resolution method and system based on latitude hierarchical reinforcement fully considers the difference in feature distribution of different latitude regions in the panorama image, divides the features of the panorama image into different regions according to latitude height through the cascaded latitude hierarchical reinforcement unit, and performs hierarchical enhancement on the features of the different regions, so that the different latitude regions are reinforced to different degrees, that is, the lower latitude region with more texture features is refined more strongly. Through this unique differential refinement of deep features, high-frequency information of different latitudes can be fully extracted, so that clearer details, textures and object edges are restored. In addition, in order to solve the information isolation of different regions caused by the spatial division, after the features of each region are merged into a complete feature map in the hierarchical reinforcement unit, a cascaded convolution layer is used to further enhance the complete feature map, promote the information interaction between different regions, avoid the occurrence of edge artifacts, and thus improve the reconstruction performance of super-resolution.

[0121] In summary, the present application effectively solves the problem of the difference in feature distribution of different latitudes in the panorama image, improves the super-resolution effect of the panorama image in different scene environments, and enhances the quality and visual experience of the panorama image.

[0122] In this document, the terms "comprises", "comprising", or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, composition of matter, or article of manufacture that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, composition of matter, or article of manufacture.

[0123] The above is a further detailed description of the present application in combination with specific preferred embodiments, and the specific implementation of the present application should not be limited to these descriptions. For those of ordinary skill in the art to which the present application belongs, without departing from the concept of the present application, a number of simple deductions or replacements can be made, which should be considered as falling within the protection scope of the present application.

Claims

1. A panorama image super-resolution method based on latitude hierarchical reinforcement, characterized in that, The method comprises the following steps: obtain a panoramic image dataset containing multiple different scenes, and divide the panoramic image dataset into a training dataset and a test dataset after preprocessing the panoramic image dataset; establish a panoramic image super-resolution model based on latitude hierarchical reinforcement, which comprises a feature extraction subnetwork, a deep feature enhancement subnetwork, an upsampling subnetwork, and a reconstruction subnetwork, wherein the feature extraction subnetwork is used to extract shallow features of an input panoramic picture; the deep feature enhancement subnetwork divides the extracted shallow features into different regions through a cascade of latitude hierarchical reinforcement units, and then performs hierarchical deep enhancement on the features of different latitude regions, extracts high-frequency information of different latitudes based on the difference in feature distribution of different latitude regions in the panoramic image, and obtains enhanced features; the upsampling subnetwork uses a subpixel convolution layer to enlarge the spatial resolution size of the enhanced features; and the reconstruction subnetwork restores the enlarged enhanced features into a high-resolution panoramic picture; train and test the panoramic image super-resolution model using the training dataset and the test dataset, respectively; implement panoramic image super-resolution using the panoramic image super-resolution model that has passed training and testing; training and testing the panoramic image super-resolution model using the training dataset and the test dataset specifically comprises: input the training dataset under multiple different scenes into the panoramic image super-resolution model in sequence; when iteratively training, calculate the loss value between the high-resolution panoramic picture generated by the panoramic image super-resolution model and the true value; train the panoramic image super-resolution model with the optimization goal of reducing the loss value, and synchronously update and optimize the network weight; evaluate the reconstruction performance of the panoramic image super-resolution model using the test dataset; the deep feature enhancement subnetwork divides the shallow features into different regions according to the height of latitude, and performs hierarchical enhancement on the features of different regions, that is, the strength of feature extraction obtained from high latitude regions to low latitude regions increases in turn; the hierarchical reinforcement unit also combines the enhanced features of each region into a feature map, and enhances the feature map using a cascade of convolution layers; the loss value is obtained by a global loss function, and the specific expression of the global loss function is: , wherein For the number of samples in the training set, 、 respectively represent the super-resolution result of the panoramic image super-resolution model on the kth input panoramic image and its corresponding ground truth, the global loss function.

2. A panoramic image super-resolution system based on latitude hierarchical reinforcement, characterized in that, The system comprises: a dataset acquisition module configured to obtain a panoramic image dataset containing multiple different scenes, and divide the panoramic image dataset into a training dataset and a test dataset after preprocessing the panoramic image dataset; The model establishing module is configured to establish a panoramic image super-resolution model based on latitude hierarchical reinforcement, which comprises a feature extraction subnetwork, a deep feature enhancement subnetwork, an upsampling subnetwork, and a reconstruction subnetwork. The feature extraction subnetwork is configured to extract shallow features of an input panoramic image. The deep feature enhancement subnetwork is configured to perform hierarchical deep enhancement on the shallow features of different latitude regions through a cascade of latitude hierarchical reinforcement units, extract high-frequency information of different latitudes based on the difference in feature distribution of different latitude regions in the panoramic image, and obtain enhanced features. The upsampling subnetwork is configured to enlarge the spatial resolution size of the enhanced features by using a sub-pixel convolution layer. The reconstruction subnetwork is configured to restore the enlarged enhanced features into high-resolution panoramic images. The model training and testing module is configured to train and test the panoramic image super-resolution model by using the training data set and the test data set, respectively. The panoramic image super-resolution implementation module is configured to implement panoramic image super-resolution by using the panoramic image super-resolution model that has passed the training and testing. The implementation process of the model training and testing module specifically comprises: The training data set in multiple different scenes is input into the panoramic image super-resolution model in sequence. When iteratively training, the loss value between the high-resolution panoramic image generated by the panoramic image super-resolution model and the true value is calculated. The panoramic image super-resolution model is trained with the optimization goal of reducing the loss value, and the network weight is updated and optimized synchronously. The reconstruction performance of the panoramic image super-resolution model is evaluated by using the test data set. The deep feature enhancement subnetwork divides the shallow features into different regions according to the latitude, and performs hierarchical enhancement on the features in different regions, that is, the intensity of feature extraction obtained from high latitude regions to low latitude regions increases in sequence. The hierarchical reinforcement unit also combines the enhanced features in each region into a feature map, and enhances the feature map by using a cascade of convolution layers. The loss value is obtained by a global loss function, and the specific expression of the global loss function is: , wherein For the number of samples in the training set, 、 respectively represent the super-resolution result of the panoramic image super-resolution model on the kth input panoramic image and its corresponding ground truth, The global loss function.

3. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps of the panoramic image super-resolution method based on latitude hierarchical reinforcement according to claim 1.

4. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The instructions are executed by the processor to implement the steps of the panoramic image super-resolution method based on latitude hierarchical reinforcement according to claim 1.

Images