Panoramic image super-resolution method, system, electronic device and storage medium

Abstract

The present invention discloses a panoramic image super-resolution method, system, electronic device and storage medium, including: establishing a panoramic image super-resolution model based on latitude hierarchical enhancement, the model including a feature extraction subnet, a deep feature enhancement subnet, an upsampling subnet and a reconstruction subnet, the feature extraction subnet is used to extract shallow features of an input panoramic image; the deep feature enhancement subnet performs hierarchical deep enhancement on shallow features of different latitude regions through cascaded latitude hierarchical enhancement units, and based on the differences in feature distribution of different latitude regions in the panoramic image, extracts high-frequency information of different latitudes to obtain enhanced features; the upsampling subnet uses a sub-pixel convolution layer to amplify the spatial resolution size of the enhanced features; the reconstruction subnet restores the amplified enhanced features into a high-resolution panoramic image; the present invention solves the problem of the differences in feature distribution of different latitudes in the panoramic image, and improves the super-resolution effect of the panoramic image under different scene environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a panoramic image super-resolution method, system, electronic device and storage medium. Background Art

[0002] With the development of panoramic image technology and the popularization of its application, people are increasingly demanding high-resolution panoramic images. Panoramic images can capture the complete view of a scene, which has wide applications in fields such as virtual reality, urban planning, architectural design, and tourism. High-resolution panoramic images can provide a 360-degree perspective, which means that every detail in the image can be clearly displayed, which is crucial for enhancing user experience and improving the practical utility of applications. For example, in virtual reality, high-resolution panoramic images can provide users with a more immersive experience, making them feel as if they are in a real environment; in urban planning, high-resolution panoramic images can help planners more accurately analyze the distribution of terrain and buildings; in the tourism industry, high-definition panoramic images can allow potential tourists to more intuitively understand the 360-degree scenery of the destination. However, compared with high-resolution ordinary pictures and videos, the shooting of high-resolution panoramic images not only requires high-precision optical equipment and image sensors for shooting, but also requires the combination and registration of pictures from different angles to construct the final 360-degree panoramic image, which leads to higher shooting costs. Especially in outdoor or complex scenes, the difficulty and cost of obtaining high-quality panoramic images will increase further.

[0003] Therefore, in the case that existing equipment cannot meet the demand for high-resolution panoramic images, the resolution of most panoramic images is relatively low, while users' demand for high-definition panoramic images is constantly increasing. This contradiction in demand has promoted the development of panoramic image super-resolution technology, through which low-resolution panoramic images can be converted into high-resolution images without changing hardware equipment, thereby meeting the actual needs of various applications.

[0004] However, panoramic images are usually stored in equirectangular projection (ERP) format, which has serious geometric distortion and latitude-related feature distribution differences. In the ERP panorama, starting from the equator, as the latitude increases toward the South Pole or the North Pole, the texture features and detail information in the panorama significantly decrease. From a mathematical point of view, ERP projection is a process from spherical projection to plane, which has serious spatial distortion. Since this stretching (also called oversampling) is usually performed by interpolation operations (such as bicubic interpolation), it will inevitably lead to smoother high-frequency information such as texture and details in high-latitude areas. Therefore, there are obvious differences in the distribution of textures and details in the ERP panorama. However, most of the existing image super-resolution methods are designed for ordinary 2D images, which cannot effectively solve this problem, and it is difficult to obtain ideal super-resolution performance and meet the visual needs of users. Summary of the invention

[0005] In view of the above problems, the present invention provides a panoramic image super-resolution method, system and storage medium based on latitude graded enhancement, aiming to restore panoramic images with higher resolution and higher quality.

[0006] According to a first aspect of an embodiment of the present disclosure, a panoramic image super-resolution method based on latitude hierarchical enhancement is provided, the method comprising the following steps:

[0007] Acquire a panoramic image dataset containing a plurality of different scenes, and divide the panoramic image dataset into a training dataset and a test dataset after preprocessing the panoramic image dataset;

[0008] A panoramic image super-resolution model based on latitude hierarchical enhancement is established, and the panoramic image super-resolution model includes a feature extraction subnet, a deep feature enhancement subnet, an upsampling subnet and a reconstruction subnet, wherein the feature extraction subnet is used to extract shallow features of an input panoramic image; the deep feature enhancement subnet performs hierarchical deep enhancement on shallow features in different latitude regions through cascaded latitude hierarchical enhancement units, and extracts high-frequency information in different latitudes based on the differences in feature distribution in different latitude regions in the panoramic image to obtain enhanced features; the upsampling subnet uses a sub-pixel convolution layer to enlarge the spatial resolution size of the enhanced features; and the reconstruction subnet restores the enlarged enhanced features into a high-resolution panoramic image;

[0009] The panoramic image super-resolution model is trained and tested using the training data set and the test data set respectively.

[0010] The panoramic image super-resolution model that has passed training and testing is used to achieve panoramic image super-resolution.

[0011] In some embodiments, using the training data set and the test data set to train and test the panoramic image super-resolution model specifically includes:

[0012] Inputting the training data sets under a plurality of different scenes into the panoramic image super-resolution model in sequence;

[0013] During iterative training, calculating a loss value between a high-resolution panoramic image generated by the panoramic image super-resolution model and a true value;

[0014] The panoramic image super-resolution model is trained with the reduction of the loss value as the optimization target, and the network weights are updated and optimized simultaneously;

[0015] The test dataset is used to evaluate the reconstruction performance of the panoramic image super-resolution model.

[0016] In some embodiments, the deep feature enhancement subnet divides the shallow features into different regions according to the latitude, and enhances the features of different regions in a hierarchical manner, that is, the intensity of feature extraction increases from high-latitude regions to low-latitude regions.

[0017] In some embodiments, the hierarchical enhancement unit further merges the enhanced features of each region into a feature map, and uses a cascaded convolutional layer to enhance the feature map.

[0018] In some embodiments, the loss value is obtained by a global loss function, and the specific expression of the global loss function is:

[0019] ,

[0020] in, is the number of samples in the training set, , They represent the panoramic image super-resolution model for the k The super-resolution result of the input panoramic image and its corresponding true value, is the global loss function.

[0021] According to a second aspect of an embodiment of the present disclosure, a panoramic image super-resolution system based on latitude hierarchical enhancement is provided, the system comprising:

[0022] A data set acquisition module, used to acquire a panoramic image data set containing a plurality of different scenes, and divide the panoramic image data set into a training data set and a test data set after preprocessing the panoramic image data set;

[0023] A model building module is used to establish a panoramic image super-resolution model based on latitude hierarchical enhancement, wherein the panoramic image super-resolution model includes a feature extraction subnet, a deep feature enhancement subnet, an upsampling subnet and a reconstruction subnet, wherein the feature extraction subnet is used to extract shallow features of an input panoramic image; the deep feature enhancement subnet performs hierarchical deep enhancement on shallow features of different latitude regions through cascaded latitude hierarchical enhancement units, and extracts high-frequency information of different latitudes based on the differences in feature distribution of different latitude regions in the panoramic image to obtain enhanced features; the upsampling subnet uses a sub-pixel convolution layer to enlarge the spatial resolution size of the enhanced features; and the reconstruction subnet restores the enlarged enhanced features into a high-resolution panoramic image;

[0024] a model training and testing module, used to train and test the panoramic image super-resolution model using the training data set and the test data set respectively,

[0025] The panoramic image super-resolution realization module is used to realize panoramic image super-resolution by using the panoramic image super-resolution model that has passed training and testing.

[0026] In some embodiments, the model training and testing module implementation process specifically includes:

[0027] Inputting the training data sets under a plurality of different scenes into the panoramic image super-resolution model in sequence;

[0028] During iterative training, calculating a loss value between a high-resolution panoramic image generated by the panoramic image super-resolution model and a true value;

[0029] The panoramic image super-resolution model is trained with the reduction of the loss value as the optimization target, and the network weights are updated and optimized simultaneously;

[0030] The test dataset is used to evaluate the reconstruction performance of the panoramic image super-resolution model.

[0031] In some embodiments, the deep feature enhancement subnet divides the shallow features into different regions according to the latitude, and enhances the features of different regions in a hierarchical manner, that is, the intensity of feature extraction increases from high-latitude regions to low-latitude regions.

[0032] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned panoramic image super-resolution method based on latitude graded enhancement are implemented.

[0033] According to a fourth aspect of an embodiment of the present disclosure, there is provided a non-temporary computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the steps of the above-mentioned panoramic image super-resolution method based on latitude graded enhancement.

[0034] The embodiments of the present disclosure provide a panoramic image super-resolution method, system and storage medium based on latitude hierarchical enhancement, and the beneficial effects thereof include:

[0035] Taking into full consideration the differences in feature distribution in different latitudes of panoramic images, the features of panoramic images are divided into different regions according to the latitude through cascaded latitude hierarchical enhancement units, and the features of these different regions are enhanced in different levels to ensure that different latitudes are enhanced to different degrees, that is, lower latitude regions with more texture features will obtain stronger refinement. Through this unique differentiated refinement of deep features, high-frequency information of different latitudes can be fully extracted, thereby restoring clearer details, textures and object edges. In addition, in order to solve the information isolation of different regions caused by this spatial division, after merging the features of each region into a complete feature map in the hierarchical enhancement unit, the cascaded convolutional layer is used to further enhance the complete feature map, promote information interaction between different regions, avoid the appearance of edge artifacts, and thus improve the reconstruction performance of super-resolution.

[0036] In summary, the present invention effectively solves the problem of the difference in the distribution of features at different latitudes in panoramic images, improves the super-resolution effect of panoramic images in different scene environments, and enhances the quality and visual experience of panoramic images.

[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention and, together with the description, serve to explain the principles of the present invention.

[0039] Figure 1 1 is a schematic diagram of a method for super-resolution of panoramic images based on latitude hierarchical enhancement in an embodiment of the present invention;

[0040] Figure 2 is a schematic diagram of the network structure of a panoramic image super-resolution model in an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of a process of a hierarchical enhancement unit in a deep feature enhancement subnetwork in an embodiment of the present invention;

[0042] Figure 4is a schematic diagram of the structure of a panoramic image super-resolution system based on latitude hierarchical enhancement in an embodiment of the present invention;

[0043] Figure 5 It is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0045] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0046] The embodiments of the present invention provide the following embodiments for a panoramic image super-resolution method, system and storage medium based on latitude hierarchical enhancement:

[0047] In the specific implementation process, the super-resolution of panoramic images is different from that of ordinary natural images. Panoramic images usually use equirectangular projection (ERP) format, which has serious geometric distortion and latitude-related texture feature differences. In the ERP panorama, the high-latitude area (north latitude) has large spatial distortion and fewer texture features; the area near the equator has small spatial distortion and more texture features; the high-latitude area (south latitude) has large spatial distortion and fewer texture features. That is, as the latitude increases toward the South Pole or the North Pole, the texture features and detail information in the panorama will obviously decrease. From a mathematical point of view, ERP projection is a process from spherical projection to plane, which suffers from severe spatial distortion and oversampling. For example, the pixels at the poles on the sphere can even be stretched to a line in the ERP plane, and this stretching and oversampling are usually performed by interpolation operations (such as bicubic interpolation and linear interpolation), which inevitably leads to smoother high-frequency information in high-latitude areas, resulting in fewer texture features. Therefore, when performing super-resolution of panoramic images, it is necessary to adapt to the spatial distortion of panoramic images and consider the differences in the distribution of texture features related to latitude. However, most existing image super-resolution technologies are oriented towards ordinary natural images and cannot achieve high-performance panoramic image super-resolution.

[0048] Based on these constraints, the key issues solved by panoramic image super-resolution methods are to achieve high-performance super-resolution of panoramic images in different input environments and to efficiently refine and enhance high-frequency information at different latitudes.

[0049] The first embodiment is used to illustrate a panoramic image super-resolution method based on latitude hierarchical enhancement, see Figure 1 , is a flow chart of the panoramic image super-resolution method based on latitude hierarchical enhancement, and the specific steps include:

[0050] S1. Acquire a panoramic image dataset containing a plurality of different scenes, and divide the panoramic image dataset into a training dataset and a test dataset after preprocessing the panoramic image dataset;

[0051] In the specific implementation process, panoramic images can be collected by different methods to build data sets for model training and testing. For example, panoramic images can be downloaded from a public and free database; keywords of different scene environments can be searched through a search engine platform, and corresponding free panoramic images can be downloaded; or the required panoramic images can be captured using mobile phones, cameras and other shooting devices.

[0052] In the specific implementation process, the panoramic picture data set must include at least 20 kinds of panoramic pictures of various environments and scenes, and then the data set is cleaned through pre-processing operations such as format unification, resolution unification, and abnormal picture deletion, and then randomly divided into a training data set (training set) and a test data set (test set). Among them, the training set accounts for 90% and is used to train the super-resolution model of the present invention, and the test set accounts for 10% and is used to evaluate the performance of the model. By inputting a low-resolution panoramic image into the super-resolution model described in the present invention, a corresponding high-resolution panoramic image can be generated.

[0053] S2. Establish a panoramic image super-resolution model based on latitude hierarchical enhancement, wherein the panoramic image super-resolution model includes a feature extraction subnet, a deep feature enhancement subnet, an upsampling subnet and a reconstruction subnet, wherein the feature extraction subnet is used to extract shallow features of the input panoramic image; the deep feature enhancement subnet performs hierarchical deep enhancement on shallow features of different latitude regions through cascaded latitude hierarchical enhancement units, and extracts high-frequency information of different latitudes based on the differences in feature distribution of different latitude regions in the panoramic image to obtain enhanced features; the upsampling subnet uses a sub-pixel convolution layer to enlarge the spatial resolution size of the enhanced features; and the reconstruction subnet restores the enlarged enhanced features into a high-resolution panoramic image;

[0054] S3, respectively using the training data set and the test data set to train and test the panoramic image super-resolution model;

[0055] S4. Implementing panoramic image super-resolution using the trained and tested panoramic image super-resolution model.

[0056] Using the training data set and the test data set to train and test the panoramic image super-resolution model specifically includes:

[0057] Inputting the training data sets under a plurality of different scenes into the panoramic image super-resolution model in sequence;

[0058] During iterative training, calculating a loss value between a high-resolution panoramic image generated by the panoramic image super-resolution model and a true value;

[0059] The panoramic image super-resolution model is trained with the reduction of the loss value as the optimization target, and the network weights are updated and optimized simultaneously;

[0060] The test dataset is used to evaluate the reconstruction performance of the panoramic image super-resolution model.

[0061] The deep feature enhancement subnet divides the shallow features into different regions according to the latitude, and performs hierarchical deep enhancement on the shallow features in different regions, that is, the intensity of feature extraction increases from high-latitude regions to low-latitude regions.

[0062] The hierarchical enhancement unit also combines the enhanced features of each region into a feature map and enhances the feature map using a cascaded convolutional layer.

[0063] During the specific implementation process, the pre-established panoramic image super-resolution model based on latitude hierarchical enhancement is initialized, and various training and testing hyperparameters required for the neural network model are set, such as the number of feature channels of the model, the number of network layers, the learning rate, the optimizer, the number of iterations, etc.

[0064] Among them, the super-resolution model of panoramic images mainly includes feature extraction subnet, feature enhancement subnet, upsampling subnet and reconstruction subnet. Figure 2 As shown, in the super-resolution model of the present invention, the feature extraction subnet uses a certain number of convolutional layers to extract shallow features of the input panoramic images under different environments; the deep feature enhancement subnet, through the cascaded latitude hierarchical enhancement units, subdivides the feature map of the panoramic image into multiple different sub-regions according to the latitude, and independently and hierarchically enhances the features of these sub-regions, so that the low-latitude regions with more texture features can be more strongly refined. Through this differentiated enhancement of deep features, the high-frequency information of each latitude can be fully refined and preserved, so as to restore clearer details, textures and edge structures in the generated high-resolution panoramic image; the upsampling subnet uses upsampling operations based on deep learning to amplify the resolution of the enhanced feature map; the reconstruction subnet restores the above-mentioned amplified features into a high-resolution panoramic image and then outputs it.

[0065] The feature extraction subnet can effectively extract the features of the input panoramic image through one or more cascaded convolutional layers, making it easier for the deep feature enhancement subnet to perform further deep and efficient enhancement.

[0066] The deep feature enhancement subnet fully considers the distribution differences of texture features related to latitude in the panoramic image, effectively discovers and refines high-frequency information at different latitudes, and then realizes efficient detail and texture restoration, thereby obtaining better panoramic image super-resolution effects. Specifically, the deep feature enhancement subnet is composed of N cascaded hierarchical enhancement units, through which differentiated enhancement of deep features can be achieved. Like the number of channels in the network, the number N of hierarchical enhancement units can be used as a hyperparameter during the training process and can be set according to actual application requirements, such as 5, 8, 10, etc. The input of the first hierarchical enhancement unit is the shallow features extracted by the feature extraction subnet, set to , and the output feature of the nth hierarchical reinforcement unit is , then the depth enhancement of the features by the cascaded N hierarchical enhancement units can be expressed as follows:

[0067] ,

[0068] in represents the function of the nth hierarchical reinforcement unit, Note that the input of the nth hierarchical reinforcement unit is the output of the n-1th hierarchical reinforcement unit.

[0069] Figure 3 The network structure and data flow of each hierarchical enhancement unit are clearly shown. The hierarchical enhancement unit first implements hierarchical enhancement of panoramic image features based on the latitude. Specifically, suppose there are J enhancement levels in each hierarchical enhancement unit, and J is also a hyperparameter pre-set during the training process. For the sake of convenience, take the nth hierarchical enhancement unit as an example, and its input feature map is , firstly, it can be divided into 2J sub-regions of the same size along the latitude, that is, , note that the feature map of the jth sub-region can be recorded as , h=H / 2L is the height of each sub-region, where H represents the feature map height.

[0070] Then, these sub-regions (also called feature maps of sub-regions) are enhanced at different levels. Considering that the spatial distortion and stretching in the panorama are symmetrical between the southern and northern hemispheres, a single level of enhancement needs to process a pair of geometrically symmetrical sub-regions at the same time. Specifically, in the j-th level of enhancement, the j-th sub-region and the (2L+1−j)-th sub-region are enhanced at the same time. .like Figure 3 As shown, in the j-th level of enhancement, a total of j convolutional layers are stacked. Therefore, different levels of enhancement have different convolutional network depths, thereby achieving hierarchical and differentiated enhancement of the feature map of the entire panoramic image. That is, as the latitude of the sub-region decreases, the more network layers used by the hierarchical enhancement unit, the higher the enhancement level, and the more obvious the enhancement for low-latitude areas with complex textures and details. This improves the adaptability to spatial distortion and oversampling in the panorama, and fully considers the differences in the distribution of texture features related to latitude in the panorama. This is because compared with high-latitude areas, low-latitude areas have more textures and details, and it is more difficult to reconstruct their super-resolution, so deeper neural networks are needed for enhancement. The above hierarchical enhancement process can be explained by the following formula:

[0071] ,

[0072] ,

[0073] ,

[0074] in, To perform the segmentation operation on the panoramic image features, is the function of the convolutional layer in the j-th level of reinforcement, and Sub-areas and The output results are enhanced in different levels.

[0075] After strengthening these sub-regions, they need to be merged again to generate a complete feature map. represents a merge operation, then the process is expressed as follows: ,

[0076] in, is the merged feature map. In addition, the residual technology is also used inside the hierarchical enhancement unit, that is, the merged feature map is connected through residual connection. With input Fusion (element-wise addition). The application of residual technology can improve training efficiency and representation ability, and avoid gradient explosion during training.

[0077] It should be noted that in this embodiment, the j-th level of enhancement can be composed of j ordinary convolutions by default. However, in order to save computing costs, it can also be set to j group convolutions to build a more efficient network model. Compared with ordinary convolutions, group convolutions can significantly reduce the amount of calculation while maintaining satisfactory performance. Secondly, a bottleneck structure design can also be adopted, using a 1 × 1 convolution before the j-th convolution layer to compress the number of channels to the original 1 / r, and finally using a 1 × 1 convolution to restore the original number of channels.

[0078] This spatial division will cause the information between different sub-regions to be isolated, affecting the modeling effect and even producing edge artifacts. In order to solve this problem, after merging the features of each region into a complete feature map in the hierarchical enhancement unit, one or more layers of cascaded convolution can be used to further enhance the complete feature map. This can promote the interaction of information between different regions, avoid the appearance of edge artifacts, and thus enhance the reconstruction performance of super-resolution. This process can be expressed by the following formula:

[0079] ,

[0080] is a function of the convolution of one or more layers of cascade. After the convolution layer is remodeled, the context information is modeled with longer-range dependencies, thereby promoting the performance of the super-resolution model and obtaining the output of the nth hierarchical enhancement unit. .

[0081] The deep enhancement subnet designs a hierarchical enhancement scheme that fully adapts to the spatial distortion of panoramic images and the differences in feature distribution caused by it. It realizes the extraction of high-frequency information at different latitudes in a spatially refined manner and effectively models the deep features of panoramic images, thereby improving the reconstruction performance of panoramic images.

[0082] The upsampling subnet is used to amplify the resolution of the enhanced features. That is, the size of the feature map is expanded to a specified multiple using deep learning-based technology (sub-pixel convolution layer) without destroying information such as object texture and details. The sub-pixel convolution layer achieves spatial resolution amplification of the feature map by mapping the number of channels to a specified multiple and then rearranging features at the same spatial position on multiple channels in the same channel.

[0083] The reconstruction subnet uses one layer of convolution to reconstruct the enlarged feature map output by the upsampling subnet into a high-definition and high-resolution panoramic image (RGB, three channels), and finally realizes a panoramic image super-resolution method based on latitude hierarchical enhancement, ensuring that the method has a good super-resolution effect on panoramic image data in different scenes and environments. Furthermore, the present invention successfully establishes a panoramic image super-resolution neural network model based on latitude hierarchical enhancement.

[0084] It should be noted that, in this embodiment, for the sake of clarity and simplicity, the feature extraction subnet, upsampling subnet and reconstruction subnet are not described in detail using corresponding formulas. However, the present invention has clearly explained the technology and details used therein, so a person skilled in the art should be able to recognize the specific functions of the subnets.

[0085] It is planned to select different panoramic images from the training set and input them into the panoramic image super-resolution neural network model based on latitude hierarchical enhancement in sequence, and implement continuous training. The input panoramic images should contain more than 20 scenes and environments, so that the adaptability of the model in different environments is significantly enhanced, thereby improving the practicality of the super-resolution method.

[0086] Select different panoramic images from the training set in turn, and iterate the execution steps continuously: input the panoramic images of different scenes in the training set into the model in turn, iterate the training model, make the loss value converge, and generate a model under complex motion scenes. In each iterative training, a certain number of panoramic images will be selected. The selected panoramic images include low-resolution panoramic images and corresponding high-resolution images as true values ​​for supervised training. The true value images and the output images of the model are input into the pre-set loss function to calculate the global loss value, and the loss value is continuously reduced through training. In this process, the network weights of the super-resolution model are continuously updated. During the iteration process, the necessity of re-adjusting parameters and training can be judged according to the change of learning rate and the decrease of loss value, and this process can be repeated. After the training is completed, the loss value should converge, and the final model can be generated.

[0087] Calculate the global loss value of the panoramic image super-resolution network. During each iteration of training, the panoramic image super-resolution network selects multiple batches of panoramic images from the data set, inputs them into the model, and calculates the loss value based on the loss function pre-set for the panoramic image super-resolution network.

[0088] The panoramic image super-resolution model based on latitude hierarchical enhancement uses the L1 Loss function as the loss function. Compared with the L2 loss, the L1 loss function is more robust, making the restored panoramic image texture and details more realistic. Moreover, the L1 loss function has better tolerance to outliers, which can improve the robustness of the model.

[0089] In order to globally optimize the super-resolution network model, the present invention needs to input the panoramic image to be super-resolution into the super-resolution model based on latitude hierarchical enhancement, and after obtaining the predicted high-resolution panoramic image, the predicted panoramic image and the corresponding true panoramic image are input into the L1 loss function to calculate the final global loss value.

[0090] In an optional implementation of this embodiment, the above calculation is based on the loss value of the super-resolution model enhanced by latitude classification, that is, the final global loss value. It can be assumed that , are the super-resolution results of the model for the k-th input panoramic image and its corresponding true value, K is the number of samples in the training set (i.e. the number of panoramic images), is the global loss function. The calculation formula of the L1 loss function mentioned in the present invention is as follows:

[0091] ,

[0092] According to the preset learning rate, the present invention sets the learning rate to 2e-4, then uses the Adaw optimizer for iterative training, and updates the weights of each subnet in the network, so that the loss value continues to decrease. When the loss value converges, the corresponding model parameters can be saved to construct a panoramic image super-resolution model based on latitude hierarchical enhancement.

[0093] According to the pre-divided data set, the panoramic images in the test set are selected to test the generated panoramic image super-resolution model based on latitude hierarchical enhancement to verify the reconstruction performance of the model. After the test is successful, it can be used as the final panoramic image super-resolution model based on latitude hierarchical enhancement.

[0094] Another embodiment is used to illustrate a panoramic image super-resolution system based on latitude hierarchical enhancement, see Figure 4 , the system 400 comprises:

[0095] The data set acquisition module 410 is used to acquire a panoramic image data set containing a plurality of different scenes, and divide the panoramic image data set into a training data set and a test data set after preprocessing the panoramic image data set;

[0096] The model building module 420 is used to establish a panoramic image super-resolution model based on latitude hierarchical enhancement, wherein the panoramic image super-resolution model includes a feature extraction subnet, a deep feature enhancement subnet, an upsampling subnet and a reconstruction subnet, wherein the feature extraction subnet is used to extract shallow features of the input panoramic image; the deep feature enhancement subnet performs hierarchical deep enhancement on shallow features of different latitude regions through cascaded latitude hierarchical enhancement units, and extracts high-frequency information of different latitudes based on the difference in feature distribution of different latitude regions in the panoramic image to obtain enhanced features; the upsampling subnet uses a sub-pixel convolution layer to enlarge the spatial resolution size of the enhanced features; and the reconstruction subnet restores the enlarged enhanced features into a high-resolution panoramic image;

[0097] The model training and testing module 430 is used to train and test the panoramic image super-resolution model using the training data set and the test data set respectively.

[0098] The panoramic image super-resolution realization module 440 is used to realize panoramic image super-resolution by using the panoramic image super-resolution model that has passed training and testing.

[0099] The model training and testing module 430 specifically includes:

[0100] Inputting the training data sets under a plurality of different scenes into the panoramic image super-resolution model in sequence;

[0101] During iterative training, calculating a loss value between a high-resolution panoramic image generated by the panoramic image super-resolution model and a true value;

[0102] The panoramic image super-resolution model is trained with the reduction of the loss value as the optimization target, and the network weights are updated and optimized simultaneously;

[0103] The test dataset is used to evaluate the reconstruction performance of the panoramic image super-resolution model.

[0104] The deep feature enhancement subnet divides the shallow features into different regions according to the latitude, and enhances the features of different regions in a hierarchical manner, that is, the intensity of feature extraction increases from high-latitude regions to low-latitude regions.

[0105] In addition to the upper module, the system 400 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present disclosure, their illustration and description are omitted here.

[0106] The other specific working processes of the panoramic image super-resolution system 400 based on latitude classification enhancement refer to the description of the above-mentioned panoramic image super-resolution method embodiment based on latitude classification enhancement, which will not be repeated here.

[0107] The panoramic image super-resolution system based on latitude hierarchical enhancement proposed in this embodiment enhances the input panoramic image across latitudes through a hierarchical enhancement mechanism, adapts to the distortion of the panoramic image, and has efficient information extraction capabilities, and has significant compatibility with panoramic images of different scenes. Therefore, the super-resolution device can achieve higher super-resolution performance of panoramic images at a lower computational cost, and can be well promoted and applied in different fields (such as virtual reality and 3D vision).

[0108] Another embodiment is used to illustrate that the system of the present invention can also be used with the help of Figure 5 The architecture of the computing device shown is implemented. Figure 5 The architecture of the computing device is shown. Figure 5 As shown, a computer system 510, a system bus 530, one or more CPUs 540, an input / output 520, a memory 550, etc. The memory 550 can store various data or files used for computer processing and / or communication and program instructions executed by the CPU including the panoramic image super-resolution method based on latitude hierarchical enhancement of the embodiment. Figure 5 The architecture shown is only exemplary and can be adjusted according to actual needs when implementing different devices. Figure 5 One or more components in. The memory 550, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the panoramic image super-resolution method based on latitude classification enhancement in the embodiment of the present invention (for example, the data set acquisition module 410, the model building module 420, the model training and testing module 430 and the panoramic image super-resolution implementation module 440 in the panoramic image super-resolution system 400 based on latitude classification enhancement). One or more CPUs 540 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions and modules stored in the memory 550, that is, to implement the above-mentioned panoramic image super-resolution method based on latitude classification enhancement, which includes the following steps:

[0109] Acquire a panoramic image dataset containing a plurality of different scenes, and divide the panoramic image dataset into a training dataset and a test dataset after preprocessing the panoramic image dataset;

[0110] A panoramic image super-resolution model based on latitude hierarchical enhancement is established, wherein the panoramic image super-resolution model includes a feature extraction subnet, a deep feature enhancement subnet, an upsampling subnet and a reconstruction subnet, wherein the feature extraction subnet is used to extract shallow features of the input panoramic image; the deep feature enhancement subnet performs hierarchical deep enhancement on shallow features of different latitude regions through cascaded latitude hierarchical enhancement units, and extracts high-frequency information of different latitudes based on the differences in feature distribution of different latitude regions in the panoramic image to obtain enhanced features; the upsampling subnet uses a sub-pixel convolution layer to enlarge the spatial resolution size of the enhanced features; and the reconstruction subnet restores the enlarged enhanced features into a high-resolution panoramic image;

[0111] The panoramic image super-resolution model is trained and tested using the training data set and the test data set respectively.

[0112] The panoramic image super-resolution model that has passed training and testing is used to achieve panoramic image super-resolution.

[0113] Of course, the processor of the server provided in the embodiment of the present invention is not limited to executing the method operations described above, but can also execute related operations in the panoramic image super-resolution method based on latitude graded enhancement provided in any embodiment of the present invention.

[0114] The memory 550 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 550 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 550 may further include a memory remotely arranged relative to one or more CPUs 540, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0115] The input / output 520 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The input / output 520 may also include a display device such as a display screen.

[0116] The embodiment of the present invention also provides a non-temporary computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor, the panoramic image super-resolution method based on latitude hierarchical enhancement described in the above embodiment is implemented. The computer-readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0117] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0118] The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the foregoing.

[0119] In addition, other specific working processes of a non-temporary computer-readable storage medium refer to the description of the above-mentioned panoramic image super-resolution method embodiment based on latitude classification enhancement, and will not be repeated here.

[0120] Based on the technical solutions provided by the above embodiments, a panoramic image super-resolution method and system based on latitude hierarchical enhancement fully considers the differences in feature distribution of different latitude regions in the panoramic image. Through the cascaded latitude hierarchical enhancement unit, the features of the panoramic image are divided into different regions according to the latitude, and the features of these different regions are graded to ensure that the different latitude regions are enhanced to different degrees, that is, the lower latitude regions with more texture features will obtain stronger refinement. Through this unique differentiated extraction of deep features, high-frequency information of different latitudes can be fully extracted, thereby restoring clearer details, textures and object edges. In addition, in order to solve the information isolation of different regions caused by this spatial division, after merging the features of each region into a complete feature map in the hierarchical enhancement unit, the cascaded convolutional layer is used to further enhance the complete feature map, promote information interaction between different regions, avoid the appearance of edge artifacts, and thus improve the reconstruction performance of super-resolution.

[0121] In summary, the present invention effectively solves the problem of the difference in the distribution of features at different latitudes in panoramic images, improves the super-resolution effect of panoramic images in different scene environments, and enhances the quality and visual experience of panoramic images.

[0122] In this document, the terms "comprises," "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such step or method.

[0123] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A panoramic image super-resolution method based on latitude hierarchical enhancement, characterized in that: The method comprises the following steps: Acquire a panoramic image dataset containing a plurality of different scenes, and divide the panoramic image dataset into a training dataset and a test dataset after preprocessing the panoramic image dataset; A panoramic image super-resolution model based on latitude hierarchical enhancement is established, and the panoramic image super-resolution model includes a feature extraction subnet, a deep feature enhancement subnet, an upsampling subnet and a reconstruction subnet, wherein the feature extraction subnet is used to extract shallow features of the input panoramic image; the deep feature enhancement subnet divides the extracted shallow features into different regions through a cascaded latitude hierarchical enhancement unit, and then performs hierarchical deep enhancement on the features of different latitude regions, and based on the difference in feature distribution of different latitude regions in the panoramic image, extracts high-frequency information of different latitudes to obtain enhanced features; the upsampling subnet uses a sub-pixel convolution layer to enlarge the spatial resolution size of the enhanced features; the reconstruction subnet restores the enlarged enhanced features into a high-resolution panoramic image; Using the training data set and the test data set to train and test the panoramic image super-resolution model respectively; Implementing panoramic image super-resolution using the trained and tested panoramic image super-resolution model; Using the training data set and the test data set to train and test the panoramic image super-resolution model specifically includes: Inputting the training data sets under a plurality of different scenes into the panoramic image super-resolution model in sequence; During iterative training, calculating a loss value between a high-resolution panoramic image generated by the panoramic image super-resolution model and a true value; The panoramic image super-resolution model is trained with the reduction of the loss value as the optimization target, and the network weights are updated and optimized simultaneously; Using the test data set to evaluate the reconstruction performance of the panoramic image super-resolution model; The deep feature enhancement subnet divides the shallow features into different regions according to the latitude, and enhances the features of different regions in different levels, that is, the intensity of feature extraction increases from high-latitude regions to low-latitude regions; The hierarchical enhancement unit also combines the enhanced features of each region into a feature map, and enhances the feature map using a cascaded convolutional layer; The loss value is obtained by the global loss function, and the specific expression of the global loss function is: , in, is the number of samples in the training set, , They represent the panoramic image super-resolution model for the k The super-resolution result of the input panoramic image and its corresponding true value, is the global loss function.

2. A panoramic image super-resolution system based on latitude hierarchical enhancement, characterized in that: The system comprises: A data set acquisition module, used to acquire a panoramic image data set containing a plurality of different scenes, and divide the panoramic image data set into a training data set and a test data set after preprocessing the panoramic image data set; A model building module is used to establish a panoramic image super-resolution model based on latitude hierarchical enhancement, wherein the panoramic image super-resolution model includes a feature extraction subnet, a deep feature enhancement subnet, an upsampling subnet and a reconstruction subnet, wherein the feature extraction subnet is used to extract shallow features of an input panoramic image; the deep feature enhancement subnet performs hierarchical deep enhancement on shallow features of different latitude regions through cascaded latitude hierarchical enhancement units, and extracts high-frequency information of different latitudes based on the differences in feature distribution of different latitude regions in the panoramic image to obtain enhanced features; the upsampling subnet uses a sub-pixel convolution layer to enlarge the spatial resolution size of the enhanced features; and the reconstruction subnet restores the enlarged enhanced features into a high-resolution panoramic image; A model training and testing module, used to train and test the panoramic image super-resolution model using the training data set and the test data set respectively; A panoramic image super-resolution realization module, used to realize panoramic image super-resolution using the trained and tested panoramic image super-resolution model; The model training and testing module implementation process specifically includes: Inputting the training data sets under a plurality of different scenes into the panoramic image super-resolution model in sequence; During iterative training, calculating a loss value between a high-resolution panoramic image generated by the panoramic image super-resolution model and a true value; The panoramic image super-resolution model is trained with the reduction of the loss value as the optimization target, and the network weights are updated and optimized simultaneously; Using the test data set to evaluate the reconstruction performance of the panoramic image super-resolution model; The deep feature enhancement subnet divides the shallow features into different regions according to the latitude, and enhances the features of different regions in a hierarchical manner, that is, the intensity of feature extraction increases from high-latitude regions to low-latitude regions; The hierarchical enhancement unit also combines the enhanced features of each region into a feature map, and enhances the feature map using a cascaded convolutional layer; The loss value is obtained by the global loss function, and the specific expression of the global loss function is: , in, is the number of samples in the training set, , They represent the panoramic image super-resolution model for the k The super-resolution result of the input panoramic image and its corresponding true value, is the global loss function.

3. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the panoramic image super-resolution method based on latitude hierarchical enhancement as claimed in claim 1 are implemented.

4. A non-transitory computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the panoramic image super-resolution method based on latitude hierarchical enhancement as claimed in claim 1 are implemented.

Images