A panoramic image enhancement processing method, device and equipment and a storage medium

Abstract

The embodiment of the present disclosure discloses a panoramic image enhancement processing method, device, equipment and storage medium, comprising obtaining a to-be-enhanced panoramic image, determining a spatial domain feature map of the to-be-enhanced panoramic image, performing frequency domain conversion on the spatial domain feature map to obtain a frequency domain feature map; performing feature enhancement processing on the frequency domain block through frequency domain perception attention to obtain a frequency domain enhanced feature map; performing image fusion on the frequency domain feature map and the frequency domain enhanced feature map, and performing inverse frequency domain conversion on the image fusion result to obtain a spatial domain enhanced image; and determining the spatial domain enhanced image as a target enhanced panoramic image. By using the method, the details features that need to be retained in the panoramic image and the features that need to be enhanced in the panoramic image are automatically determined through frequency domain perception attention, so that the influence of artifact distortion on the enhancement of the panoramic image can be better avoided, and the content of the region of interest can be enriched and the features can be enhanced. In this way, the immersive experience of the panoramic image is improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a panoramic image enhancement processing method, apparatus, device, and storage medium. Background Technology

[0002] Panoramic images cover an entire 360-degree spatial scene and are often called spherical images. The wide viewpoint range of panoramic images provides viewers with an immersive visual experience. However, existing panoramic images often suffer from low resolution, noise, or blurriness due to limitations in image acquisition equipment or transmission bandwidth. To improve the image quality of panoramic images, image enhancement processing is needed to increase resolution, remove noise, or deblur and improve image sharpness.

[0003] Existing enhancement methods primarily target two-dimensional images. Therefore, before enhancing panoramic images, it is necessary to project the panoramic image from a sphere onto a plane using an equidistant cylindrical projection method to form an equidistant cylindrical image of the panoramic image. However, this projection method introduces artifact distortion problems; that is, the polar regions of the sphere are distorted due to the projection, resulting in severe distortion in the image regions at both ends of the equidistant cylindrical image formed by the projection.

[0004] Currently, no enhancement processing method for panoramic images has been found to effectively avoid the above problems. Summary of the Invention

[0005] This disclosure provides a panoramic image enhancement processing method, apparatus, device, and storage medium, which can effectively enhance panoramic images and avoid the impact of artifact distortion caused by existing panoramic image projection on image enhancement.

[0006] In a first aspect, embodiments of this disclosure provide a panoramic image enhancement processing method, the method comprising:

[0007] The spatial domain feature map is transformed into a frequency domain feature map.

[0008] Frequency domain enhanced feature maps are obtained by performing feature enhancement processing on frequency domain blocks in the frequency domain feature map through frequency domain aware attention.

[0009] Image fusion is performed on the frequency domain feature map and the frequency domain enhanced feature map to obtain an image fusion result, and the image fusion result is subjected to inverse frequency domain transformation to obtain a spatial domain enhanced image;

[0010] The spatial domain enhanced image is determined as the target enhanced panoramic image of the panoramic image to be enhanced.

[0011] Secondly, embodiments of this disclosure also provide a panoramic image enhancement processing apparatus, the apparatus comprising:

[0012] A feature determination module is used to acquire the panoramic image to be enhanced, and to determine the spatial domain feature map of the panoramic image to be enhanced.

[0013] The frequency domain conversion module is used to perform frequency domain conversion on the spatial domain feature map to obtain a frequency domain feature map;

[0014] The feature enhancement module is used to perform feature enhancement processing on the frequency blocks in the frequency domain feature map through frequency domain aware attention to obtain a frequency domain enhanced feature map;

[0015] The image fusion module is used to perform image fusion on the frequency domain feature map and the frequency domain enhanced feature map to obtain an image fusion result, and to perform inverse frequency domain transformation on the image fusion result to obtain a spatial domain enhanced image;

[0016] The target determination module is used to determine the spatial domain enhanced image as the target enhanced panoramic image of the panoramic image to be enhanced.

[0017] Thirdly, embodiments of this disclosure also provide a computer device, the computer device comprising:

[0018] One or more processors;

[0019] Storage device for storing one or more programs.

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the panoramic image enhancement processing method provided in any embodiment of this disclosure.

[0021] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the panoramic image enhancement processing method provided in any embodiment of this disclosure.

[0022] Fifthly, embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements the panoramic image enhancement processing method provided in any embodiment of this disclosure.

[0023] This disclosure provides a panoramic image enhancement processing method, apparatus, device, and storage medium. The method first acquires a panoramic image to be enhanced and determines a spatial domain feature map of the panoramic image. Then, it performs a frequency domain transformation on the spatial domain feature map to obtain a frequency domain feature map. Next, it performs feature enhancement processing on the frequency domain blocks in the frequency domain feature map using frequency domain-aware attention to obtain a frequency domain-enhanced feature map. Afterward, it performs image fusion on the frequency domain feature map and the frequency domain-enhanced feature map to obtain an image fusion result, and performs an inverse frequency domain transformation on the image fusion result to obtain a spatial domain-enhanced image. Finally, it determines the spatial domain-enhanced image as the target enhanced panoramic image of the panoramic image to be enhanced. This embodiment, by converting the panoramic image to the frequency domain, can automatically determine the detailed features that need to be retained and the features that need to be enhanced in the panoramic image through frequency domain-aware attention in the feature map represented in the frequency domain. This enables feature reconstruction of some regions in the panoramic image to be enhanced, thereby better modulating the artifact distortion caused by the change from sphere to plane projection in the panoramic image to be enhanced. Meanwhile, the frequency-domain perceptual attention employed in this technical solution ensures rich content and enhanced features in the region of interest within the panoramic image to be enhanced, thereby improving the immersive experience of the panoramic image. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the exemplary embodiments of this disclosure, the accompanying drawings used in describing the embodiments are briefly introduced below. Obviously, the accompanying drawings described are only a portion of the embodiments to be described in this disclosure, and not all of them. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort.

[0025] Figure 1 A schematic flowchart of a panoramic image enhancement processing method provided in an embodiment of this disclosure;

[0026] Figure 2 This is a schematic diagram of the structure of a panoramic image enhancement processing device provided in an embodiment of the present disclosure;

[0027] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0028] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0029] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0030] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0031] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules, or units, and are not used to limit the order of functions performed by these devices, modules, or units or their interdependencies. It should also be noted that the modifications of "a" and "a plurality of" mentioned in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0032] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0033] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0034] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0035] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0036] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0037] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0038] Figure 1 This is a flowchart illustrating a panoramic image enhancement processing method provided in an embodiment of the present disclosure. This embodiment is applicable to situations where panoramic images are enhanced. The method can be executed by a panoramic image enhancement processing device, which can be implemented by software and / or hardware and can be configured in a terminal and / or server to implement the panoramic image enhancement processing method in this embodiment of the present disclosure.

[0039] It should be noted that one application scenario of this embodiment can be described as follows: When performing feature enhancement on the image content of a panoramic image, the enhancement is mainly performed on the panoramic image formed by isochoric projection. This projected panoramic image suffers from artifact distortion. When conventional image enhancement methods are used to enhance the panoramic image, this artifact distortion problem affects the processing effect. Furthermore, when users view panoramic images, they often focus on the area of ​​interest (POI). Existing image enhancement methods focus on processing the entire image, affecting the quality improvement of the POI area and consequently impacting user satisfaction with the panoramic image quality.

[0040] Based on this, this embodiment provides a panoramic image enhancement processing method, thereby proposing a more suitable method for enhancing panoramic images and improving the image quality after enhancement. Specifically, as shown... Figure 1 As shown, the panoramic image enhancement processing method provided in this embodiment may include:

[0041] S101. Obtain the panoramic image to be enhanced, and determine the spatial domain feature map of the panoramic image to be enhanced.

[0042] In this embodiment, the panoramic image to be enhanced can be considered as a panoramic image that needs enhancement processing after projecting a spherical image representing a 360-degree spatial scene using an isocillant projection method. The enhancement points of the panoramic image to be enhanced may include resolution enhancement processing, noise reduction enhancement processing, or deblurring enhancement processing, etc. It should be noted that the method provided in this embodiment can perform enhancement processing on different enhancement points of the panoramic image to be enhanced, but each execution of the method only performs enhancement processing on any single enhancement point.

[0043] In this embodiment, the spatial domain feature map can be specifically understood as the feature map obtained by extracting features from the panoramic image to be enhanced in the spatial domain. This spatial domain feature map can be determined by a network model used for feature extraction. For example, feature extraction of the panoramic image to be enhanced can be achieved using network models with convolutional network structures and network models with multiple encoder-decoder network structures, respectively. The feature map output after feature extraction from the panoramic image to be enhanced using the convolutional network structure can be further input into the network model with multiple encoder-decoder network structures, and the feature extraction result output by the network model can then be used as the spatial frequency domain feature map.

[0044] The example given above is equivalent to feature extraction through two network structures, which can achieve deeper feature extraction of the panoramic image to be enhanced with a relatively small network structure size.

[0045] S102. Perform frequency domain transformation on the spatial domain feature map to obtain a frequency domain feature map.

[0046] In this embodiment, this step transforms the features of the panoramic image to be enhanced to the frequency domain, converting the spatial domain feature map of the panoramic image to be enhanced into a series of frequency-based feature blocks. This refines the features of the panoramic image, allowing for subsequent fusion processing of feature blocks from different frequency bands to reduce the negative impact of projection distortion, noise, or blurring on the panoramic image during enhancement. For example, as a frequency domain transformation implementation, a discrete cosine transform can be used to transform the spatial domain feature map of the panoramic image to be enhanced to the frequency domain, forming a frequency domain feature map of the panoramic image to be enhanced.

[0047] It should be noted that the spatial domain feature map participating in the frequency domain conversion in this embodiment can be considered to have undergone resolution sampling processing in advance. Specifically, an upsampling network structure can be used to upsample the spatial domain feature map to ensure that the spatial domain feature map has the set target resolution. The resolution upsampling of the spatial domain feature map in this embodiment can effectively reduce the interference of image resolution on the enhancement processing of panoramic images in the frequency domain.

[0048] S103. Perform feature enhancement processing on the frequency domain blocks in the frequency domain feature map through frequency domain awareness attention to obtain a frequency domain enhanced feature map.

[0049] In this embodiment, the features of the panoramic image can be represented in the form of frequency domain blocks in the frequency domain feature map formed after frequency domain transformation. This step can capture the spatial correlation of features in the frequency domain feature map through frequency domain awareness attention, and realize deep feature fusion of panoramic images across latitudes and frequency bands. This ensures that each frequency band in the frequency domain representation of each frequency block is treated equally, thereby better focusing on and effectively preserving the high-frequency detail texture regions in the panoramic image.

[0050] As described above, this step, through frequency domain perceptual attention, can also guide the generation of texture features in the high-frequency band by using low-frequency feature information in the low-frequency band of the panoramic image. This achieves the effect of distinguishing between textures with artifact distortion and regular textures, thereby avoiding the impact of artifact distortion features on panoramic image enhancement. This effectively enhances the detail features in the panoramic image, and finally obtains the frequency domain enhanced feature map based on frequency domain perceptual attention enhancement processing in the frequency domain.

[0051] It can be seen that, compared to the frequency domain feature map, this frequency domain enhanced feature map retains the high-frequency detail texture features in the panoramic image, and can also effectively generate high-frequency texture features based on low-frequency band feature information. Essentially, the frequency domain enhanced feature map presents more high-frequency detail features of the panoramic image, thereby enhancing the texture representation within the viewpoint of interest range of the panoramic image.

[0052] S104. Perform image fusion on the frequency domain feature map and the frequency domain enhanced feature map to obtain an image fusion result, and perform inverse frequency domain transformation on the image fusion result to obtain a spatial domain enhanced image.

[0053] In this embodiment, the frequency domain enhanced feature map better enhances the texture details within the viewpoint of interest range of the panoramic image. Fusing this frequency domain enhanced feature map with the frequency domain feature map is equivalent to feature fusion of the panoramic image, enabling effective feature reconstruction of the panoramic image. Simultaneously, converting the image fusion result to the spatial domain ensures that the reconstructed panoramic image is presented at the target resolution with improved resolution in the spatial domain. In this embodiment, the image presented in the spatial domain can be denoted as the spatial domain enhanced image.

[0054] S105. The spatial domain enhanced image is determined as the target enhanced panoramic image of the panoramic image to be enhanced.

[0055] In this embodiment, the enhancement processing of the panoramic image is mainly achieved through frequency domain aware attention after conversion to the frequency domain. The result of the enhancement processing is then fused with the initial frequency domain feature map after frequency domain conversion to achieve feature reconstruction of the panoramic image. This step can directly determine the spatial domain enhanced image after feature reconstruction and conversion to the spatial domain as the target enhanced panoramic image.

[0056] This embodiment provides a panoramic image enhancement processing method. By converting the panoramic image to the frequency domain, frequency-domain perceptual attention can be used in the feature map represented in the frequency domain to automatically determine the detailed features that need to be retained and the features that need to be enhanced. This enables feature reconstruction of certain regions in the panoramic image to be enhanced, thereby better modulating the artifact distortion caused by the change from sphere to plane projection. Simultaneously, the frequency-domain perceptual attention used in this technical solution also ensures the richness of content and feature enhancement in the regions of interest within the panoramic image to be enhanced. This improves the immersive experience of panoramic images.

[0057] As a first optional embodiment of this embodiment, based on the above embodiment, the spatial domain feature map of the panoramic image to be enhanced can be further optimized as follows: through the feature extraction sub-model in the target enhancement processing model, the panoramic image to be enhanced is subjected to feature extraction and upsampling processing to obtain the spatial domain feature map output by the feature extraction sub-model.

[0058] In this embodiment, the enhancement processing of the panorama to be enhanced can be considered to be mainly achieved through a target enhancement processing model. The target enhancement processing model may include sub-models for performing different enhancement processing logics. For example, it may include a feature extraction sub-model for feature processing of the panorama image to be enhanced, or it may include a frequency-domain aware attention sub-model for frequency-domain awareness of the panorama image to be enhanced in the frequency domain.

[0059] As one implementation method, this embodiment can use a feature extraction sub-model included in the target enhancement processing model to perform feature extraction and resolution upsampling processing on the panoramic image to be enhanced. Specifically, the feature extraction sub-model can include multiple network structures for feature extraction. For example, it can include a relatively simple convolutional network structure, or a relatively complex multi-encoder / decoder serial network structure. Different network structures can be used to achieve feature extraction at different levels from the panoramic image to be enhanced.

[0060] For example, the first-level feature extraction can be considered as using a relatively simple convolutional network structure to extract features from the panoramic image to be enhanced. Because the network structure is relatively simple, it's equivalent to using a shallower feature extraction method, thus obtaining a first-level feature extraction map. Then, a more complex feature extraction network composed of multiple encoder-decoder networks can be used to perform a second-level, deeper feature extraction on the feature extraction map obtained after the first-level shallow feature extraction, resulting in the second-level feature extraction map. This two-level feature extraction method can achieve deeper feature extraction from the panoramic image to be enhanced while ensuring the network size remains within an appropriate range.

[0061] As described above, this embodiment can also enhance the resolution of the feature extraction map by using the upsampling network structure included in the feature extraction sub-model, so that the resolution of the feature extraction map of the panoramic image to be enhanced can be improved to the target resolution. This embodiment is equivalent to obtaining a spatial domain feature map after the resolution reaches the target resolution.

[0062] Specifically, based on this first optional embodiment, as one way to determine the spatial domain feature map, the process of extracting features and upsampling the panoramic image to be enhanced through the feature extraction sub-model in the target enhancement processing model to obtain the spatial domain feature map output by the feature extraction sub-model can be further optimized to the following steps:

[0063] a1) The first-level feature extraction is performed on the panoramic image to be enhanced by the convolutional network structure in the feature extraction sub-model to obtain the first-level feature extraction map.

[0064] In this embodiment, this step can be regarded as the first-level feature extraction of the panoramic image to be enhanced. Specifically, the first-level feature extraction of the panoramic image to be enhanced in the RGB spatial domain, which is the input data, can be performed through a relatively simple convolutional network structure, thereby obtaining the first-level feature extraction map output by the convolutional network structure.

[0065] As described above, extracting more detailed features from the panoramic image to be enhanced requires deeper feature mining and extraction. However, directly performing deeper feature mining and extraction on the panoramic image requires constructing a large-scale network model. Given limited computing resources and space, it is difficult to directly provide a large-scale network model for computation. Therefore, this embodiment considers multi-level feature extraction from the panoramic image to be enhanced.

[0066] In this embodiment, a relatively simple convolutional network structure provided in this step can be used to perform shallow feature extraction on the panoramic image to be enhanced. This shallow feature extraction is equivalent to a coarse extraction of features from the panoramic image to be enhanced, and cannot reveal more detailed features.

[0067] b1) Through multiple encoding and decoding network structures in the feature extraction sub-model, the first-layer feature extraction map is subjected to second-level feature extraction to obtain a second-layer feature extraction map, wherein the feature extraction depth of the second layer is greater than that of the first layer.

[0068] Following the above description, to obtain more detailed feature information from the panoramic image to be enhanced, this step requires deeper feature extraction. Specifically, this step can be seen as the second-level feature extraction of the panoramic image to be enhanced. The multiple encoder-decoder network structures it provides are equivalent to providing a more complex deep learning network structure. Through this more complex deep learning network structure, a second-level deep feature extraction can be performed on the first-level feature extraction map used as input data, thereby obtaining the second-level feature extraction map output by the deep learning network structure. In this embodiment, multiple serial encoder-decoder network structures can be used to form this deep learning network structure.

[0069] It can be seen that the second-layer feature extraction map is obtained by further extracting features based on the first-layer feature extraction map through multiple encoding and decoding network structures with more complex structures and deeper network layers. Therefore, the second-layer feature extraction performed in this step can be considered as a more refined feature extraction of the panoramic image to be enhanced, and thus the feature extraction depth of the second-layer feature extraction in this step can be considered to be greater than the feature extraction depth of the first layer.

[0070] c1) By using the upsampling network structure in the feature extraction sub-model, the resolution of the second layer feature extraction map is upsampled to the target resolution to obtain a spatial domain feature map with the target resolution.

[0071] In this embodiment, the feature extraction sub-model also includes an upsampling network structure to improve the resolution of the panoramic image to be enhanced, and finally obtain a spatial domain feature map of the specific target resolution.

[0072] The target resolution can be set differently depending on the application scenario. When performing super-resolution enhancement on a panoramic image, the target resolution can be set to the product of the original resolution of the panoramic image and the super-resolution ratio. Alternatively, when performing denoising or deblurring enhancement on a panoramic image, the target resolution can be set to the original resolution of the panoramic image.

[0073] The above-described technical solution in this embodiment provides a specific implementation of the spatial domain feature map. The spatial domain feature map determined by the above technical solution can be regarded as the basic data support for subsequent enhancement processing of panoramic images in the frequency domain, so as to realize the conversion of panoramic images from the spatial domain to the frequency domain.

[0074] As a second optional embodiment of this embodiment, based on the above embodiment, the frequency domain transformation of the spatial domain feature map to obtain the frequency domain feature map can be further specified as follows: the spatial domain feature map is transformed into a frequency domain feature map including a first set number of frequency domain blocks by means of discrete cosine transform.

[0075] In this embodiment, the obtained spatial domain feature map is mainly used to achieve the conversion to the frequency domain. One conversion method can be the discrete cosine transform, which can convert the continuous feature map in the spatial domain into discrete frequency domain blocks divided according to frequency bands in the frequency domain, and specifically, it can convert to obtain a first set number of frequency domain blocks.

[0076] Specifically, the formula for the discrete cosine transform can be described as follows:

[0077]

[0078] Where F(u,v) represents the function for transforming the spatial domain feature map to the frequency domain, after which a frequency domain feature map containing a first predetermined number of frequency domain blocks of size N×N can be formed; the first predetermined number can be expressed as C×H×W, where C is the number of feature channels, H is the height of the frequency domain feature map, and W is the width of the frequency domain feature map; f(i,j) represents the spatial domain feature map, u,v∈[0,N-1] are the frequency domain coordinates, and c(.) is the normalization operation used to satisfy orthogonality. When u,v equals 0, c(u) and c(v) can take any value. When u and v are not equal to 0, c(u) and c(v) can take any values.

[0079] The above-described technical solution in this embodiment, through the frequency domain conversion method, is equivalent to converting the panoramic image to be enhanced in the spatial domain into a series of frequency-based frequency domain blocks, providing data support for subsequent fusion processing of different frequency bands. This method of representing panoramic images in the frequency domain also effectively reduces the negative impacts of panoramic image projection distortion and noise blurring on panoramic image enhancement.

[0080] As a third optional embodiment of this embodiment, based on the above embodiment, the frequency domain block in the frequency domain feature map can be enhanced by frequency domain-aware attention to obtain a frequency domain enhanced feature map. This can be further specified as follows: by using the frequency domain-aware attention sub-model in the target enhancement processing model, the spatial correlation results between the frequency domain blocks included in the frequency domain feature map are determined, and the frequency domain blocks are enhanced by the spatial correlation results to obtain the enhanced frequency domain feature map.

[0081] In this embodiment, frequency domain-aware attention processing can be performed on the frequency domain feature map using the frequency domain-aware attention sub-model in the target enhancement processing model. Frequency domain-aware attention can be used to determine the spatial correlation of frequency domain blocks in the frequency domain feature map, and feature enhancement of frequency domain blocks can be achieved based on the spatial correlation results between different frequency domain blocks. Feature enhancement of the frequency domain feature map in this embodiment includes preserving and focusing on high-frequency detail texture regions in the panoramic image, as well as effectively generating high-frequency texture features based on low-frequency information in the panoramic image.

[0082] As one implementation method, spatial correlation results can be determined by calculating the similarity between frequency domain blocks. After normalization and weighting of the spatial correlation results, a normalized weight matrix representing the correlation between frequency domain blocks can be obtained. Through this normalized weight matrix, it is possible to determine which are the high-frequency detail features of the panoramic image, thereby effectively preserving the high-frequency detail features. It is also possible to determine which are the low-frequency information of the panoramic image, thereby using the low-frequency information to generate the high-frequency texture features of the corresponding frequency domain blocks, thus achieving panoramic image reconstruction with higher quality texture features. The above operation is also equivalent to realizing feature enhancement of the frequency domain feature map.

[0083] Specifically, based on the third optional embodiment described above, as one implementation of the frequency domain enhanced feature map, the spatial correlation results between the frequency domain blocks included in the frequency domain feature map can be determined through the frequency domain-aware attention sub-model in the target enhancement processing model, and the frequency domain blocks can be enhanced using the spatial correlation results to obtain the enhanced frequency domain feature map. This can be further optimized into the following steps:

[0084] a2) Using the frequency domain awareness attention sub-model, the first set number of frequency domain blocks included in the frequency domain feature map are subdivided into a second set number of frequency domain units.

[0085] In this embodiment, it can be considered that this step and the following steps for feature enhancement are all implemented through a frequency domain-aware attention sub-model. The frequency domain feature map can be used as input data and input to the frequency domain-aware attention sub-model. The frequency domain-aware attention sub-model can first subdivide a first set number of frequency domain blocks contained in the frequency domain feature map, so that the frequency domain blocks are refined to more fine-grained frequency domain units, wherein the frequency domain unit can be used as the smallest unit of frequency domain dimension.

[0086] For example, using N*N as the existing frequency domain block size, C can also be used as the existing number of feature channels in the frequency domain feature map, and H and W can be used as the height and width of the frequency domain feature map. This step refines the frequency domain feature map to a finer granularity, equivalent to refining an N*N frequency domain block of size C×H×W to a second predetermined number of frequency domain units, where the second predetermined number can be N. 2 ×H / NM×W / NM, and the size of the resulting frequency domain unit can be 1×1×C×M. 2 .

[0087] b2) Determine the query vector, key vector, and numerical vector for each frequency domain unit, and determine the similarity value between the two frequency domain units based on the query vector and key vector.

[0088] In this embodiment, this step can be considered as the self-attention mechanism calculation performed by the frequency domain-aware attention sub-model for each frequency domain unit. This calculation determines the similarity value between two frequency domain units, thereby determining the spatial correlation of frequency domain blocks. Specifically, the self-attention mechanism calculation includes three vector calculation formulas: a query vector calculation formula, a key vector calculation formula, and a numerical vector calculation formula. Using these formulas, the query vector, key vector, and numerical vector for each frequency domain unit can be determined. Then, based on the query vector and key vector of each frequency domain unit, the similarity value between pairs of frequency domain units can be determined.

[0089] c2) The normalized similarity values ​​are weighted using the numerical vector to obtain a normalized weight matrix that characterizes the spatial correlation of each frequency domain block.

[0090] In this embodiment, this step can also be considered as the processing logic of the frequency domain-aware attention sub-model. Specifically, the similarity values ​​of the two frequency domain units determined in the above steps can be normalized. The normalized similarity values ​​can be weighted by multiplying them with numerical vectors. Then, the activation function in the frequency domain-aware attention sub-model is used to activate the weighted result, thereby obtaining the normalized weight matrix that represents the spatial correlation of each frequency domain block.

[0091] d2) The frequency domain block is enhanced using the normalized weight matrix to obtain the enhanced frequency domain feature map.

[0092] In this embodiment, the normalized weight matrix obtained above can provide data support for feature enhancement. Through this normalized weight matrix, the differences between artifact features and regular texture features of panoramic images under projection conditions can be effectively distinguished. Thus, the normalized weight matrix can better focus on the high-frequency detail features of panoramic images and better preserve the high-frequency detail features. At the same time, it can also generate high-frequency texture features for texture enhancement of panoramic images based on the low-frequency features of panoramic images.

[0093] The above-described technical solution in this embodiment provides a specific implementation of feature enhancement of the frequency domain feature map of a panoramic image through frequency domain perceptual attention. Feature enhancement of the panoramic image in the frequency domain can effectively distinguish between artifact features and regular texture features in a panoramic image under projection distortion, thereby avoiding enhancement interference caused by artifacts generated by projection distortion. The feature enhancement in this embodiment is equivalent to feature reconstruction of the panoramic image, which better improves the feature quality of the panoramic image and ensures the feature enhancement effect of the panoramic image.

[0094] As a fourth optional embodiment of this example, based on the above embodiments, image fusion can be performed on the frequency domain feature map and the frequency domain enhanced feature map to obtain an image fusion result, and the image fusion result can be subjected to inverse frequency domain transformation to obtain a spatial domain enhanced image, which can be further optimized as follows:

[0095] a3) The frequency domain enhancement feature map is convolved by the image fusion sub-model in the target enhancement processing model, and the frequency domain convolution enhancement feature map after the convolution operation is fused with the frequency domain feature map through residual connection to obtain the frequency domain fusion enhancement feature map as the image fusion result.

[0096] In this embodiment, the frequency domain enhanced feature map determined by frequency domain aware attention mainly realizes feature enhancement of the local viewpoint of interest region. In order to ensure the authenticity of the panoramic image, it is also necessary to perform image fusion with the frequency domain enhanced feature map that has not been enhanced, so as to add more image features of the panoramic image.

[0097] For example, this embodiment can optimize the target enhancement processing model by including an image fusion sub-model, which can be used to achieve image fusion of the frequency domain enhancement feature map and the frequency domain feature map. Specifically, a simple convolutional network structure in the image fusion sub-model can be used to perform convolution calculations on the frequency domain enhancement feature map. The convolutional network structure can output the frequency domain convolutional enhancement feature map after the convolution operation. Then, the frequency domain convolutional enhancement feature map and the frequency domain feature map can be used as inputs, and the residual network structure in the image fusion sub-model can be used to perform residual connections between the frequency domain convolutional enhancement feature map and the frequency domain feature map to achieve image fusion. The image fusion result after residual connection can be determined as the frequency domain fused enhancement feature map.

[0098] b3) The frequency domain fusion enhancement feature map is inversely converted to the spatial domain using discrete cosine transform to obtain a spatial domain enhanced image.

[0099] This step can use the inverse discrete cosine transform (ICT) to inversely transform the enhanced panoramic image from the frequency domain to the spatial domain. This yields a spatial domain enhanced image after the inverse transform.

[0100] The inverse discrete cosine transform (ICT) also has a corresponding formula, which can be expressed as follows:

[0101]

[0102] Where f(i,j) represents the function transformation expression of a frequency domain block of size N×N from the frequency domain to the spatial domain, which can characterize the spatial domain feature map, i and j are spatial domain coordinates, F(u,v) represents the frequency domain feature map, u,v∈[0,N-1] are frequency domain coordinates, and c(.) is the normalization operation used to satisfy orthogonality. When u,v equals 0, c(u) and c(v) can take any value. When u and v are not equal to 0, c(u) and c(v) can take any values.

[0103] This embodiment provides a specific implementation of further feature fusion and inverse transformation of the frequency domain enhanced feature map to the spatial domain to obtain a spatial domain enhanced feature map. The obtained spatial domain enhanced feature map can be used as the target enhanced panoramic image in the final panoramic image enhancement processing. Compared with the enhancement map determined by existing technologies, the obtained spatial domain enhanced feature map can better modulate the artifact distortion caused by the change from sphere to plane projection in the panoramic image to be enhanced, and can also ensure the richness of content and feature enhancement of the region of interest in the panoramic image to be enhanced. This improves the immersive experience of panoramic images.

[0104] As a fifth optional embodiment of this example, based on the above embodiments, the training steps of the target enhancement processing model can be further specified as follows:

[0105] In this embodiment, the target enhancement processing model can be considered to include a feature extraction sub-model, a frequency domain awareness attention sub-model, and an image fusion sub-model. Therefore, training the target enhancement processing model can be considered as training each sub-model included in the target enhancement processing model.

[0106] a4) Obtain a sample training set and an initial enhancement processing model, wherein the sample training set includes at least one sample tuple, the sample tuple includes a sample panoramic image and a sample enhanced panoramic image of the sample panoramic image.

[0107] In this embodiment, the initial augmentation model can be obtained through this step, and a sample training set for model training can be acquired. The sample training set contains sample pairs specifically determined for model training. Each sample pair includes a sample panoramic image and its corresponding augmented panoramic image. These are high-quality panoramic images that are not limited by transmission bandwidth or storage space, and are genuinely acquired. This high quality is mainly reflected in the absence of noise, blur, and high resolution. It should also be noted that these augmented panoramic images are also panoramic images after equidistant cylindrical projection.

[0108] Correspondingly, the sample panoramic image can be considered as a high-quality panoramic image that has been degraded from a real-world acquisition. For example, the degradation processing may include low-resolution processing, noise addition processing, or blurring processing. The panoramic image formed after degrading the high-quality panoramic image can be used as the sample panoramic image required in this embodiment. In this embodiment, a certain size of sample tuples can be used to construct the sample training set.

[0109] In this embodiment, the initial enhancement processing model can be considered as a pre-constructed network model that includes the network structure of each sub-model, and each sub-model in the initial enhancement processing model can be considered to have initial network parameters.

[0110] b4) Input the sample panoramic image into the initial enhancement processing model to obtain the actual enhanced panoramic image of the sample panoramic image.

[0111] In this embodiment, the process of training the target enhancement processing model can be regarded as an iterative training process. In each iteration, one or more sample pairs can be selected from the sample training set to participate in the model training. Specifically, the panoramic images of the selected sample pairs can be used as input data to the initial enhancement processing model.

[0112] It is understood that the initial enhancement processing model can sequentially enhance the sample panoramic image according to its constituent sub-models. For example, the feature extraction sub-model in the initial enhancement processing model can be used to extract features from the sample panoramic image and perform frequency domain transformation, obtaining the sample frequency domain feature map output by the feature extraction sub-model. Then, the sample frequency domain feature map can be used as input data to the frequency domain perception sub-model, which performs perceptual attention processing on the sample frequency domain feature map in the frequency domain, obtaining the sample frequency domain enhanced feature map output by the frequency domain perception sub-model.

[0113] As described above, the obtained sample frequency domain feature map and sample frequency domain enhanced feature map can be used as input to the image fusion sub-model. The image fusion sub-model performs image fusion on the sample frequency domain feature map and sample frequency domain enhanced feature map, and transforms the fused image from the frequency domain to the spatial domain through inverse transformation. The resulting sample spatial domain enhanced image can be used as the actual enhanced panoramic image output by the initial enhancement processing model. This actual enhanced panoramic image can be considered as the enhancement processing result obtained after one iteration.

[0114] c4) Based on the sample enhanced panoramic image and the actual enhanced panoramic image, determine the first loss function value of the first loss function, the second loss function value of the second loss function, and the third loss function value of the third loss function.

[0115] In this embodiment, the iterative training process of the target enhancement processing model can be considered as a process of reverse adaptive adjustment of network parameters, and the loss function value can be used as the information value required for reverse adaptive adjustment. This embodiment can achieve the determination of the loss function value required for reverse adaptive adjustment of network parameters through this step and step d4) below.

[0116] Analysis reveals that the initial enhancement processing model comprises multiple sub-models, each with different network structures and processing logic. To achieve effective training of each sub-model, this embodiment considers setting different loss functions from multiple perspectives. For example, three loss functions can be designed, each focusing on the training of a different model. The weighted sum of the loss function values ​​corresponding to the different loss functions, as described in the following steps, yields the final target loss function that adapts the network parameters.

[0117] As one implementation of this step, determining the first loss function value of the first loss function, the second loss function value of the second loss function, and the third loss function value of the third loss function based on the enhanced panoramic image of the sample and the actual enhanced panoramic image can be further specified as the following steps:

[0118] c41) Using the first loss function, the average absolute value of the first pixel difference is determined based on the first pixel value of each pixel in the sample enhanced panoramic image and the second pixel value of each pixel in the actual enhanced panoramic image.

[0119] In this embodiment, the first loss function can be set to be related to the pixel values ​​of each pixel in the two panoramic images (sample enhanced panoramic image and actual enhanced panoramic image). Specifically, the first pixel value of each pixel in the sample enhanced panoramic image and the second pixel value of each pixel in the actual enhanced panoramic image can be determined. Then, the difference between two pixel values ​​with equal pixel coordinates can be obtained to obtain the pixel difference value corresponding to each pixel coordinate. Finally, the average absolute value of the pixel difference values ​​of all pixel coordinates can be calculated to obtain the average absolute value of the first pixel difference. This average absolute value of the first pixel difference can be used as the first loss function value of the first loss function.

[0120] c42) Using the second loss function, based on the third pixel value of each pixel in the first residual image and the fourth pixel value of each pixel in the second residual image, the average absolute value of the second pixel difference is determined as the value of the second loss function. The first residual image and the second residual image are obtained by performing Gaussian blur processing on the sample enhanced panoramic image and the actual enhanced panoramic image, respectively.

[0121] In this embodiment, the second loss function can be set to be correlated with the images after Gaussian blurring of the two panoramic images (sample enhanced panoramic image and actual enhanced panoramic image). Specifically, the images after Gaussian blurring of the two panoramic images can be denoted as the first residual image and the second residual image, respectively. In this step, the third pixel value of each pixel in the first residual image and the fourth pixel value of each pixel in the second residual image can be obtained. Then, the difference between two pixel values ​​with the same pixel coordinates can be obtained to obtain the pixel difference value corresponding to each pixel coordinate. Finally, the average absolute value of the pixel difference values ​​of all pixel coordinates can be calculated to obtain the average absolute value of the second pixel difference. This average absolute value of the second pixel difference can be used as the second loss function value of the second loss function.

[0122] c43) Using the third loss function, based on the fifth pixel value of each pixel in the first frequency domain image and the sixth pixel value of each pixel in the second frequency domain image, the average absolute value of the third pixel difference is determined as the value of the third loss function. The first frequency domain image and the second frequency domain image are obtained by performing frequency domain transformation processing on the sample enhanced panoramic image and the actual enhanced panoramic image, respectively.

[0123] In this embodiment, the third loss function can be set to be correlated with the images after frequency domain transformation of the two panoramic images (sample enhanced panoramic image and actual enhanced panoramic image). Specifically, the frequency domain images after frequency domain transformation of the two panoramic images can be denoted as the first frequency domain image and the second frequency domain image, respectively. In this step, the third pixel value of each pixel in the first frequency domain image and the fourth pixel value of each pixel in the second frequency domain image can be obtained. Then, the difference between two pixel values ​​with the same pixel coordinates can be obtained to obtain the pixel difference value corresponding to each pixel coordinate. Finally, the average absolute value of the pixel difference values ​​of all pixel coordinates can be calculated to obtain the average absolute value of the third pixel difference. This average absolute value of the third pixel difference can be used as the third loss function value of the third loss function.

[0124] d4) Determine the target loss function value based on the first loss function value, the second loss function value, and the third loss function value.

[0125] In this embodiment, the first loss function value, the second loss function value, and the third loss function value are determined through the above steps. This step can be used to weight these three loss function values. The weight value corresponding to each loss function value can be set empirically; specifically, a relatively high weight can be given to the first loss function value. Finally, the weighted calculation result can be used as the target loss function value in this embodiment.

[0126] e4) Update the network parameters of each sub-model included in the initial augmentation processing model according to the target loss function value, and return to re-execute step b4) until the training end condition is met, and determine the initial augmentation processing model corresponding to the end of training as the target augmentation processing model.

[0127] In this embodiment, the target loss function value determined above can be used as a learning metric and fed back to each sub-model of the initial enhancement processing model through gradient descent or gradient ascent, so as to achieve adaptive adjustment of the network parameters of each sub-model.

[0128] The adjusted initial enhancement model is equivalent to completing one iteration. Since the training termination condition has not been met, it is necessary to return to step b4 and perform a new round of iteration. This process is repeated until the training termination condition is met. The initial enhancement model obtained after training can be considered as a target enhancement model that can participate in the enhancement processing of panoramic images involved in practical applications.

[0129] The above technical solution in this embodiment provides a training implementation of the target enhancement processing model. The target enhancement processing model formed through training can better achieve effective enhancement processing of panoramic images.

[0130] Figure 2This is a schematic diagram of a panoramic image enhancement processing device provided in an embodiment of this disclosure. This embodiment is applicable to situations where panoramic images are enhanced. The device can be implemented by software and / or hardware and can be configured in a terminal and / or server to implement the panoramic image enhancement processing method in this embodiment of the disclosure. Specifically, the device may include: a feature determination module 21, a frequency domain conversion module 22, a feature enhancement module 23, an image fusion module 24, and a target determination module 25.

[0131] The feature determination module 21 is used to acquire the panoramic image to be enhanced, and to determine the spatial domain feature map of the panoramic image to be enhanced.

[0132] Frequency domain conversion module 22 is used to perform frequency domain conversion on the spatial domain feature map to obtain a frequency domain feature map;

[0133] Feature enhancement module 23 is used to perform feature enhancement processing on the frequency blocks in the frequency domain feature map through frequency domain aware attention to obtain a frequency domain enhanced feature map;

[0134] The image fusion module 24 is used to perform image fusion on the frequency domain feature map and the frequency domain enhanced feature map to obtain an image fusion result, and to perform inverse frequency domain transformation on the image fusion result to obtain a spatial domain enhanced image;

[0135] The target determination module 25 is used to determine the spatial domain enhanced image as the target enhanced panoramic image of the panoramic image to be enhanced.

[0136] This embodiment provides a panoramic image enhancement processing device. By converting the panoramic image to the frequency domain, frequency-domain perceptual attention can be used in the feature map to automatically determine the detailed features that need to be retained and the features that need to be enhanced in the panoramic image. This enables feature reconstruction of a portion of the panoramic image to be enhanced, thereby better modulating the artifact distortion caused by the change from sphere to plane projection in the panoramic image to be enhanced. Simultaneously, the frequency-domain perceptual attention used in this technical solution also ensures the richness of content and feature enhancement in the region of interest in the panoramic image to be enhanced. This improves the immersive experience of panoramic images.

[0137] Furthermore, the feature determination module 21 is used to perform feature extraction and upsampling processing on the panoramic image to be enhanced through the feature extraction sub-model in the target enhancement processing model, so as to obtain the spatial domain feature map output by the feature extraction sub-model.

[0138] Furthermore, the feature determination module 21 can specifically be used for:

[0139] The first-level feature extraction is performed on the panoramic image to be enhanced using the convolutional network structure in the feature extraction sub-model to obtain the first-level feature extraction map.

[0140] Through multiple encoder-decoder network structures in the feature extraction sub-model, the first-layer feature extraction map is subjected to second-layer feature extraction to obtain a second-layer feature extraction map, wherein the feature extraction depth of the second layer is greater than that of the first layer.

[0141] The resolution of the second-layer feature extraction map is upsampled to the target resolution using the upsampling network structure in the feature extraction sub-model, thereby obtaining a spatial domain feature map with the target resolution.

[0142] Furthermore, the frequency domain conversion module 22 can be specifically used to convert the spatial domain feature map into a frequency domain feature map including a first predetermined number of frequency domain blocks through discrete cosine transform.

[0143] Furthermore, the feature enhancement module 23 may specifically include: a frequency domain sensing enhancement unit,

[0144] The frequency domain sensing enhancement unit is used to determine the spatial correlation results between the frequency domain blocks included in the frequency domain feature map through the frequency domain sensing attention sub-model in the target enhancement processing model, and to perform feature enhancement on the frequency domain blocks through the spatial correlation results to obtain the enhanced frequency domain feature map.

[0145] Furthermore, the frequency domain sensing enhancement unit can specifically be used for:

[0146] The frequency domain sensing attention sub-model is used to subdivide a first set number of frequency domain blocks included in the frequency domain feature map into a second set number of frequency domain units.

[0147] Determine the query vector, key vector, and numerical vector for each frequency domain unit, and determine the similarity value between two frequency domain units based on the query vector and key vector.

[0148] The normalized similarity values ​​are weighted using the numerical vectors to obtain a normalized weight matrix that characterizes the spatial correlation of each frequency domain block.

[0149] The frequency domain block is enhanced using the normalized weight matrix to obtain the enhanced frequency domain feature map.

[0150] Furthermore, the image fusion module 24 can specifically be used for:

[0151] The frequency domain enhanced feature map is convolved by the image fusion sub-model in the target enhancement processing model, and the frequency domain convolution enhanced feature map after the convolution operation is fused with the frequency domain feature map through residual connection to obtain the frequency domain fused enhanced feature map as the image fusion result.

[0152] The frequency domain fusion enhancement feature map is inversely converted to the spatial domain using discrete cosine transform to obtain a spatial domain enhanced image.

[0153] Furthermore, the device also includes a model training module for training the target augmentation processing model;

[0154] The model training module may specifically include:

[0155] An acquisition unit is used to acquire a sample training set and an initial enhancement processing model, wherein the sample training set includes at least one sample tuple, the sample tuple includes a sample panoramic image and a sample enhanced panoramic image of the sample panoramic image.

[0156] An input unit is used to input the sample panoramic image into the initial enhancement processing model to obtain the actual enhanced panoramic image of the sample panoramic image.

[0157] The first determining unit is used to determine the first loss function value of the first loss function, the second loss function value of the second loss function, and the third loss function value of the third loss function based on the sample enhanced panoramic image and the actual enhanced panoramic image.

[0158] The second determining unit is used to determine the target loss function value based on the first loss function value, the second loss function value, and the third loss function value;

[0159] The self-learning unit is used to update the network parameters of each sub-model included in the initial augmentation processing model according to the target loss function value, and return to re-execute the step of inputting the sample panoramic image into the initial augmentation processing model until the training termination condition is met, and the initial augmentation processing model corresponding to the training termination is determined as the target augmentation processing model.

[0160] Furthermore, the first determining unit can specifically be used for:

[0161] Obtain a sample training set and an initial enhancement processing model, wherein the sample training set includes at least one sample pair, the sample pair includes a sample panoramic image and a sample enhanced panoramic image of the sample panoramic image;

[0162] The sample panoramic image is input into the initial enhancement processing model to obtain the actual enhanced panoramic image of the sample panoramic image;

[0163] Based on the sample enhanced panoramic image and the actual enhanced panoramic image, determine the first loss function value of the first loss function, the second loss function value of the second loss function, and the third loss function value of the third loss function;

[0164] The target loss function value is determined based on the first loss function value, the second loss function value, and the third loss function value.

[0165] Update the network parameters of each sub-model included in the initial augmentation model according to the target loss function value, return to re-execute the step of inputting the sample panoramic image into the initial augmentation model, until the training termination condition is met, and determine the initial augmentation model corresponding to the training termination as the target augmentation model.

[0166] The above-described apparatus can execute the methods provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the methods.

[0167] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0168] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Reference is made below. Figure 3 It illustrates a computer device suitable for implementing embodiments of the present disclosure (e.g., Figure 3 The diagram below shows the structure of the terminal device or server 30. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 3 The computer device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0169] like Figure 3As shown, the computer device 30 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 31, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 32 or a program loaded from a storage device 38 into a random access memory (RAM) 33. The RAM 33 also stores various programs and data required for the operation of the computer device 30. The processing unit 31, the ROM 32, and the RAM 33 are interconnected via a bus 35. An edit / output (I / O) interface 34 is also connected to the bus 35.

[0170] Typically, the following devices can be connected to I / O interface 34: input devices 36 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 37 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 38 including, for example, magnetic tapes, hard disks, etc.; and communication devices 39. Communication device 39 allows computer device 30 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 3 A computer device 30 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0171] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 39, or installed from a storage device 38, or installed from a ROM 32. When the computer program is executed by the processing device 31, it performs the functions defined in the methods of embodiments of this disclosure.

[0172] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0173] The computer device provided in this embodiment and the panoramic image enhancement processing method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0174] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the panoramic image enhancement processing method provided in the above embodiments.

[0175] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0176] In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0177] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0178] The aforementioned computer-readable medium may be included in the aforementioned computer device; or it may exist independently and not assembled into the computer device.

[0179] The aforementioned computer-readable medium carries one or more programs that, when executed by the computer device, cause the computer device to:

[0180] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0181] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0182] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0183] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0184] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0185] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0186] Furthermore, although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while some specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0187] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A panoramic image enhancement processing method, characterized in that, include: Acquire the panoramic image to be enhanced, and determine the spatial domain feature map of the panoramic image to be enhanced; The spatial domain feature map is transformed into a frequency domain feature map. Frequency domain enhanced feature maps are obtained by performing feature enhancement processing on frequency domain blocks in the frequency domain feature map through frequency domain aware attention. Image fusion is performed on the frequency domain feature map and the frequency domain enhanced feature map to obtain an image fusion result, and the image fusion result is subjected to inverse frequency domain transformation to obtain a spatial domain enhanced image; The spatial domain enhanced image is determined as the target enhanced panoramic image of the panoramic image to be enhanced.

2. The method according to claim 1, characterized in that, Determining the spatial domain feature map of the panoramic image to be enhanced includes: The feature extraction sub-model in the target enhancement processing model is used to extract features and perform upsampling on the panoramic image to be enhanced, thereby obtaining the spatial domain feature map output by the feature extraction sub-model.

3. The method according to claim 2, characterized in that, The step involves using a feature extraction sub-model within the target enhancement processing model to extract features from the panoramic image to be enhanced and perform upsampling processing to obtain a spatial domain feature map output by the feature extraction sub-model. This includes: The first-level feature extraction is performed on the panoramic image to be enhanced by the convolutional network structure in the feature extraction sub-model to obtain the first-level feature extraction map. Through multiple encoder-decoder network structures in the feature extraction sub-model, the first-layer feature extraction map is subjected to second-layer feature extraction to obtain a second-layer feature extraction map, wherein the feature extraction depth of the second layer is greater than that of the first layer. The resolution of the second-layer feature extraction map is upsampled to the target resolution using the upsampling network structure in the feature extraction sub-model, thereby obtaining a spatial domain feature map with the target resolution.

4. The method according to claim 1, characterized in that, The step of performing frequency domain transformation on the spatial domain feature map to obtain a frequency domain feature map includes: The spatial domain feature map is converted into a frequency domain feature map including a first predetermined number of frequency domain blocks by using discrete cosine transform.

5. The method according to claim 1, characterized in that, The step of performing feature enhancement processing on frequency blocks in the frequency domain feature map through frequency domain-aware attention to obtain a frequency domain enhanced feature map includes: By using the frequency domain-aware attention sub-model in the target enhancement processing model, the spatial correlation results between the frequency domain blocks included in the frequency domain feature map are determined, and the frequency domain blocks are enhanced using the spatial correlation results to obtain the enhanced frequency domain feature map.

6. The method according to claim 5, characterized in that, The step involves determining the spatial correlation results between frequency blocks included in the frequency domain feature map using the frequency domain-aware attention sub-model in the target enhancement processing model, and then performing feature enhancement on the frequency blocks based on the spatial correlation results to obtain a feature-enhanced frequency domain feature map, including: The frequency domain sensing attention sub-model is used to subdivide a first set number of frequency domain blocks included in the frequency domain feature map into a second set number of frequency domain units. Determine the query vector, key vector, and numerical vector for each frequency domain unit, and determine the similarity value between two frequency domain units based on the query vector and key vector. The normalized similarity values ​​are weighted using the numerical vectors to obtain a normalized weight matrix that characterizes the spatial correlation of each frequency domain block. The frequency domain block is enhanced using the normalized weight matrix to obtain the enhanced frequency domain feature map.

7. The method according to claim 1, characterized in that, The step of fusing the frequency domain feature map and the frequency domain enhanced feature map to obtain an image fusion result, and then performing an inverse frequency domain transform on the image fusion result to obtain a spatial domain enhanced image, includes: The frequency domain enhanced feature map is convolved by the image fusion sub-model in the target enhancement processing model, and the frequency domain convolution enhanced feature map after the convolution operation is fused with the frequency domain feature map through residual connection to obtain the frequency domain fused enhanced feature map as the image fusion result. The frequency domain fusion enhancement feature map is inversely converted to the spatial domain using discrete cosine transform to obtain a spatial domain enhanced image.

8. The method according to any one of claims 2-7, characterized in that, The training steps of the target enhancement processing model include: Obtain a sample training set and an initial enhancement processing model, wherein the sample training set includes at least one sample tuple, the sample tuple includes a sample panoramic image and a sample enhanced panoramic image of the sample panoramic image; The sample panoramic image is input into the initial enhancement processing model to obtain the actual enhanced panoramic image of the sample panoramic image; Based on the sample enhanced panoramic image and the actual enhanced panoramic image, determine the first loss function value of the first loss function, the second loss function value of the second loss function, and the third loss function value of the third loss function; The target loss function value is determined based on the first loss function value, the second loss function value, and the third loss function value. Update the network parameters of each sub-model included in the initial augmentation model according to the target loss function value, return to re-execute the step of inputting the sample panoramic image into the initial augmentation model, until the training termination condition is met, and determine the initial augmentation model corresponding to the training termination as the target augmentation model.

9. The method according to claim 8, characterized in that, Based on the sample enhanced panoramic image and the actual enhanced panoramic image, the first loss function value of the first loss function, the second loss function value of the second loss function, and the third loss function value of the third loss function are determined, including: Using the first loss function, the average absolute value of the first pixel difference is determined based on the first pixel value of each pixel in the sample enhanced panoramic image and the second pixel value of each pixel in the actual enhanced panoramic image. The average absolute value of the second pixel difference is determined by the second loss function based on the third pixel value of each pixel in the first residual image and the fourth pixel value of each pixel in the second residual image. The first residual image and the second residual image are obtained by performing Gaussian blur processing on the sample enhanced panoramic image and the actual enhanced panoramic image, respectively. The average absolute value of the third pixel difference is determined by the third loss function based on the fifth pixel value of each pixel in the first frequency domain image and the sixth pixel value of each pixel in the second frequency domain image. The first frequency domain image is obtained by performing frequency domain transformation on the sample enhanced panoramic image, and the second frequency domain image is obtained by performing frequency domain transformation on the actual enhanced panoramic image.

10. A panoramic image enhancement processing device, characterized in that, include: A feature determination module is used to acquire the panoramic image to be enhanced, and to determine the spatial domain feature map of the panoramic image to be enhanced. The frequency domain conversion module is used to perform frequency domain conversion on the spatial domain feature map to obtain a frequency domain feature map; The feature enhancement module is used to perform feature enhancement processing on the frequency blocks in the frequency domain feature map through frequency domain aware attention to obtain a frequency domain enhanced feature map; The image fusion module is used to perform image fusion on the frequency domain feature map and the frequency domain enhanced feature map to obtain an image fusion result, and to perform inverse frequency domain transformation on the image fusion result to obtain a spatial domain enhanced image; The target determination module is used to determine the spatial domain enhanced image as the target enhanced panoramic image of the panoramic image to be enhanced.

11. A computer device, characterized in that, The computer device includes: One or more processors; a storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the panoramic image enhancement processing method as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the panoramic image enhancement processing method as described in any one of claims 1-9.

13. A computer program product comprising a computer program that, when executed by a processor, implements the panoramic image enhancement processing method according to any one of claims 1-9.

Images