A method and device for drawing light and shadow of virtual objects based on light source position

CN113538704BActive Publication Date: 2024-11-01HISENSE VISUAL TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110788416.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-07-13
Publication Date
2024-11-01
Estimated Expiration
2041-07-13

AI Technical Summary

Technical Problem

目前,大多基于深度学习网络预测光源的位置,然后基于预测的光源位置绘制虚拟对象的光影,但计算过程较为复杂,且需要预先训练深度模型,对设备的配置要求较高

Benefits of technology

[0023] In the above embodiment of the present application, cluster analysis is performed on the HDR panoramic binary image obtained after preprocessing of the HDR panoramic image, and each centroid point in the classification cluster is used as at least one light source corresponding to the virtual object in the real scene, and determined According to the image coordinates of the corresponding light source, according to the panoramic image projection format, the image coordinates of at least one light source are converted into three-dimensional coordinates in the real scene. The position of the light source in the real scene is determined through the image features of the HDR panoramic image. The calculation process is simple. , Further, according to the three-dimensional coordinates of at least one light source, the light and shadow of the virtual object are drawn, so that the virtual object and the real scene maintain lighting consistency, and the realism of the integration of the virtual object and the real scene is improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113538704B_ABST
    Figure CN113538704B_ABST
Patent Text Reader

Abstract

The present application relates to the field of AR technology, and provides a method and device for drawing light and shadow of a virtual object based on the position of a light source, specifically comprising: when drawing the light and shadow of a virtual object, obtaining a high dynamic range HDR panoramic image; preprocessing the HDR panoramic image to obtain an HDR panoramic binary image; performing cluster analysis on the HDR panoramic binary image to obtain image coordinates of at least one light source corresponding to the virtual object in a real scene; according to a panoramic image projection format, converting the image coordinates of the at least one light source into three-dimensional coordinates in the real scene, determining the position of the light source in the real scene through the image features of the HDR panoramic image, and having a simple calculation process. Furthermore, according to the three-dimensional coordinates of the at least one light source, drawing the light and shadow of the virtual object, so that the virtual object maintains lighting consistency with the real scene, thereby improving the sense of reality of the fusion of the virtual object and the real scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of augmented reality (AR) technology, and in particular to a method and device for drawing light and shadow of a virtual object based on the position of a light source. Background Art

[0002] AR technology is a technology that cleverly integrates virtual objects with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. It simulates computer-generated virtual objects such as text, three-dimensional models, and audio, and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

[0003] In AR technology, in order to make virtual objects more naturally integrated with real scenes and improve the realism of AR effects, it is necessary to ensure the lighting consistency between virtual objects and real scenes. Lighting consistency reflects the correct lighting matching relationship between virtual objects and real scenes, such as light and shadow, to achieve the effect of enhancing the consistency between AR and real scenes.

[0004] The position of the light source in lighting consistency determines the light and shadow effects of virtual objects. At present, most of them predict the position of the light source based on the deep learning network, and then draw the light and shadow of the virtual object based on the predicted light source position. However, the calculation process is relatively complicated and requires pre-training of the deep model, which has high requirements on the configuration of the device. Summary of the invention

[0005] The embodiments of the present application provide a method and device for drawing light and shadow of a virtual object based on the position of a light source, so as to improve the realism of the fusion of the virtual object and the real scene.

[0006] In a first aspect, an embodiment of the present application provides a method for drawing light and shadow of a virtual object based on a light source position, comprising:

[0007] When drawing the light and shadow of virtual objects, obtain high dynamic range HDR panoramic images;

[0008] Preprocessing the HDR panoramic image to obtain an HDR panoramic binary image;

[0009] Performing cluster analysis on the HDR panoramic binary image to obtain image coordinates of at least one light source corresponding to the virtual object in the real scene;

[0010] According to the panoramic image projection format, the image coordinates of the at least one light source are converted into three-dimensional coordinates in the real scene respectively;

[0011] The light and shadow of the virtual object are drawn according to the three-dimensional coordinates of the at least one light source.

[0012] In a second aspect, an embodiment of the present application provides a device for drawing light and shadow of a virtual object based on a light source position, including a display, a communication interface, a memory, and a processor;

[0013] The display is connected to the processor and configured to display an augmented reality (AR) image;

[0014] The communication interface is connected to the processor and is configured to send and receive data;

[0015] The memory is connected to the processor and configured to store computer program instructions;

[0016] The processor is configured to perform the following operations according to the computer program instructions:

[0017] When drawing the light and shadow of virtual objects, obtain high dynamic range HDR panoramic images;

[0018] Preprocessing the HDR panoramic image to obtain an HDR panoramic binary image;

[0019] Performing cluster analysis on the HDR panoramic binary image to obtain image coordinates of at least one light source corresponding to the virtual object in the real scene;

[0020] According to the panoramic image projection format, the image coordinates of the at least one light source are converted into three-dimensional coordinates in the real scene respectively;

[0021] The light and shadow of the virtual object are drawn according to the three-dimensional coordinates of the at least one light source.

[0022] In a third aspect, the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method for drawing light and shadow of a virtual object based on the position of a light source provided in an embodiment of the present application.

[0023] In the above-mentioned embodiment of the present application, a cluster analysis is performed on the HDR panoramic binary image obtained after HDR panoramic image preprocessing, each centroid point in the classification cluster is used as at least one light source corresponding to the virtual object in the real scene, and the image coordinates of the corresponding light source are determined. According to the panoramic image projection format, the image coordinates of the at least one light source are respectively converted into three-dimensional coordinates in the real scene, and the position of the light source in the real scene is determined by the image features of the HDR panoramic image. The calculation process is simple. Furthermore, according to the three-dimensional coordinates of the at least one light source, the light and shadow of the virtual object are drawn, so that the virtual object maintains lighting consistency with the real scene, thereby improving the realism of the fusion of the virtual object and the real scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0025] Figure 1 The following is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0026] Figure 2 A flowchart of a method for drawing light and shadow of a virtual object based on the position of a light source provided in an embodiment of the present application is exemplarily shown;

[0027] Figure 3 The ERP projection format diagram provided in the embodiment of the present application is exemplarily shown;

[0028] Figure 4 The mapping relationship diagram between the latitude and longitude coordinate system and the world coordinate system provided by the embodiment of the present application is exemplarily shown;

[0029] Figure 5 The structure diagram of a device for drawing light and shadow of a virtual object based on the position of a light source provided in an embodiment of the present application is exemplified. DETAILED DESCRIPTION

[0030] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0031] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of the disclosure can also constitute a complete implementation method separately.

[0032] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0033] In addition, the terms "include" and "have" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0034] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0035] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0036] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application is shown; Figure 1 As shown, through human-computer interaction, AR technology is used to superimpose a teapot 101 on the basis of the real scene, and the teapot is a virtual object. Figure 1 As shown, there is a light source on the left side of the real scene. In order to make the virtual teapot 101 more naturally integrated with the objects (1021 to 1024) in the real scene, it is necessary to draw corresponding light and shadow for the virtual teapot 101 according to the position of the light source, thereby improving the realism of the virtual teapot in the real scene.

[0037] It should be noted that Figure 1 This is just an example. The light source in the embodiment of the present application can be located inside the captured real scene image or outside the real scene image. Moreover, after the light source in the real scene is irradiated onto the real object, a virtual light source can also be formed based on the reflection of light.

[0038] Figure 2 A flowchart of a method for drawing light and shadow of a virtual object based on the position of a light source provided in an embodiment of the present application is exemplarily shown; Figure 2 As shown, the process can be performed by a device that draws light and shadow of a virtual object based on the position of a light source, and mainly includes the following steps:

[0039] S201: When drawing light and shadow of a virtual object, an HDR panoramic image is obtained.

[0040] In this step, when drawing light and shadow for the virtual object superimposed on the real scene, a high-dynamic range (HDR) panoramic image of the real scene is collected by a panoramic camera.

[0041] S202: Preprocess the HDR panoramic image to obtain an HDR panoramic binary image.

[0042] In this step, the HDR panoramic image is converted into an HDR panoramic grayscale image. The embodiment of the present application does not impose any restrictive requirements on the conversion method, including but not limited to taking the average of the three channels of R, G, and B as the grayscale value, or taking the pixel value of any one of the three channels of R, G, and B as the grayscale value, or weighting the pixel values ​​of the three channels of R, G, and B according to preset pixel weights, and taking the weighted pixel value as the grayscale value.

[0043] After conversion to an HDR panoramic grayscale image, the pixel values ​​of the pixels at the light source position in the HDR panoramic grayscale image are higher, and the pixel values ​​of the pixels at the non-light source position are lower. Further, the HDR panoramic grayscale image is adaptively thresholded to obtain an HDR panoramic binary image. Specifically, the pixel value in the HDR panoramic grayscale image is compared with the adaptive threshold. If the pixel value is less than the adaptive threshold, the pixel value is set to 0, otherwise the pixel value is set to 1, and an HDR panoramic binary image represented by 0 and 1 is obtained.

[0044] In order to smooth the contours of objects in the real scene, an opening operation is performed on the HDR panoramic binary image to obtain a clean HDR panoramic binary image. Based on the clean HDR panoramic binary image, the position of the light source in the real scene is determined.

[0045] S203: Perform cluster analysis on the HDR panoramic binary image to obtain image coordinates of at least one light source corresponding to the virtual object in the real scene.

[0046] In this step, the K-mean algorithm is used to perform cluster analysis on the HDR panoramic binary image. Specifically, the K centroid points of the clusters are initialized. Considering that the real scene may contain multiple light sources, K is set to be greater than or equal to 1. According to the distance between each pixel point and each centroid point in the HDR panoramic binary image, the cluster where each pixel point is located is determined, and the centroid point of each cluster is re-determined until the preset conditions are met. Clustering is stopped, and the position of the centroid point is the position of the light source. Among them, the preset conditions can be set according to actual needs. For example, it can be set to reach the maximum number of iterations or the centroid point of the re-determined cluster no longer changes.

[0047] After clustering, the two-dimensional coordinates of K centroid points are obtained, recorded as (m, n). According to the two-dimensional coordinates of the K centroid points, the image coordinates of the K light sources corresponding to the virtual object in the real scene can be obtained.

[0048] S204: According to the panoramic image projection format, image coordinates of at least one light source are converted into three-dimensional coordinates in the real scene.

[0049] In this step, there is no restrictive requirement on the panoramic image projection format, including any one of the Equi-Rectangular Projection (ERP) format, the Equal-area Projection (EAP) format, and the Cubemap Projection (CMP) format.

[0050] Among them, ERP is the most widely used panoramic image projection method. This projection method maps the longitude of the earth into equally spaced vertical lines and the latitude of the earth into equally spaced horizontal lines, which can generate a two-dimensional image with an aspect ratio of 2:1, thereby establishing a mapping relationship between the two-dimensional image and the three-dimensional space. Figure 3 A schematic diagram of an equirectangular projection is shown as an example.

[0051] In the embodiment of the present application, the ERP projection format is taken as an example. For any light source i among the at least one light source, the three-dimensional coordinates thereof are determined as follows:

[0052] According to the two-dimensional coordinates (m, n) of the centroid point i corresponding to the light source i and the pixel width W and pixel height H of the HDR panoramic image, the image coordinates (u, v) of the light source i are determined. The calculation formula is as follows:

[0053]

[0054] According to the conversion relationship between the image coordinate system and the longitude and latitude coordinate system, the image coordinates (u, v) of light source i are converted into longitude and latitude coordinates. The calculation formula is as follows:

[0055]

[0056] According to the conversion relationship between the longitude and latitude coordinate system and the three-dimensional space coordinate system, such as Figure 4 As shown, the longitude and latitude coordinates of light source i Converted to three-dimensional coordinates (X, Y, Z), the calculation formula is as follows:

[0057]

[0058] S205: Drawing light and shadow of the virtual object according to the three-dimensional coordinates of at least one light source.

[0059] In this step, the illumination angle corresponding to the virtual object can be determined according to the three-dimensional coordinates of at least one light source, and light and shadow are drawn for the superimposed virtual object according to the illumination angle.

[0060] In some embodiments, considering that there is more than one light source in a real scene, it is necessary to consider the impact of the superposition of multiple light sources on the light and shadow effects of virtual objects. In specific implementations, the distance between at least one light source and the virtual object is determined according to the three-dimensional coordinates of at least one light source; the light and shadow weight of the corresponding light source is set according to the distance corresponding to each of the at least one light source. The closer the distance, the greater the light and shadow weight, and the farther the distance, the smaller the light and shadow weight, that is, the light and shadow weight is positively correlated with the distance; the light and shadow of the virtual object is drawn according to the light and shadow weight of each of the at least one light source.

[0061] In other embodiments, when there are multiple light sources in a real scene, considering that the illumination intensity of each light source is different, the influence of a light source with a smaller illumination intensity on the light and shadow effect of a virtual object can be ignored. The illumination intensity can be determined by the number of pixels included in the classification cluster.

[0062] In a specific implementation, for any one light source i among at least one light source, the number of pixels with a preset pixel value (the value is 1 in the embodiment of the present application) contained in the classification cluster to which the centroid point i corresponding to the light source i belongs is determined, and the determined number of pixels is compared with the preset pixel threshold. If the number of pixels is less than the preset pixel threshold, it indicates that the light source i has little influence on the light and shadow effect of the virtual object and can be ignored. In this case, the image coordinates of the light source i are eliminated, thereby improving the efficiency of drawing the light and shadow of the virtual object.

[0063] In the above-mentioned embodiments of the present application, considering the influence of the light source position in the real scene on the pixel value size in the HDR panoramic image collected by the panoramic camera, that is, the grayscale difference between the light source position and other parts is relatively large, the HDR panoramic grayscale image is adaptively thresholded to obtain an HDR panoramic binary image, and then the HDR panoramic binary image is clustered by the K-meas algorithm, and the image coordinates of the light source are determined according to the two-dimensional coordinates of the centroid point. Finally, the three-dimensional coordinates of the light source in the real world coordinate system are determined by 3601ib (panoramic image projection format). Compared with the use of a deep learning network to predict the light source position, the computational complexity is lower, and the requirements for device configuration are reduced. Furthermore, based on the determined position coordinates of the light source, the light and shadow of the virtual object are drawn, so that the virtual object and the real scene are more naturally integrated, and the authenticity of the fusion is improved.

[0064] Based on the same technical concept, an embodiment of the present application provides a device for drawing the light and shadow of a virtual object based on the position of a light source. The device can be a display terminal with interactive functions such as a smart TV, a smart phone, a laptop computer, a desktop computer, a VR device, an AR device, etc. The device can implement the method of drawing the light and shadow of a virtual object based on the position of the light source in the embodiment of the present application, and can achieve the same technical effect, which will not be repeated here.

[0065] See also Figure 5The device includes a display 501, a communication interface 502, a memory 503, and a processor 504. The display 501, the communication interface 502, and the memory 503 are connected to the processor 504 via a bus (in Figure 5 The display 501 is configured to display an AR image; the communication interface 502 is configured to send and receive data; the memory 503 is configured to store computer program instructions; and the processor 504 is configured to perform the following operations according to the computer program instructions:

[0066] When drawing the light and shadow of virtual objects, obtain high dynamic range HDR panoramic images;

[0067] Preprocessing the HDR panoramic image to obtain an HDR panoramic binary image;

[0068] Performing cluster analysis on the HDR panoramic binary image to obtain image coordinates of at least one light source corresponding to the virtual object in the real scene;

[0069] According to the panoramic image projection format, the image coordinates of at least one light source are converted into three-dimensional coordinates in the real scene;

[0070] The light and shadow of the virtual object are drawn according to the three-dimensional coordinates of at least one light source.

[0071] Optionally, the processor 504 is further configured to:

[0072] For any one of the at least one light source, determine the number of pixels of the preset pixel value contained in the classification cluster to which the centroid point corresponding to the light source belongs;

[0073] If the number of pixels is less than the preset pixel threshold, the image coordinates of the light source are discarded.

[0074] Optionally, the processor 504 is specifically configured to:

[0075] Determining the distance between at least one light source and the virtual object according to the three-dimensional coordinates of at least one light source;

[0076] According to the distance corresponding to at least one light source, a light and shadow weight of the corresponding light source is set, and the light and shadow weight is positively correlated with the distance;

[0077] The light and shadow of the virtual object are rendered according to the light and shadow weight of the at least one light source.

[0078] Optionally, the processor 504 is specifically configured to:

[0079] Convert HDR panoramic images into HDR panoramic grayscale images;

[0080] Perform adaptive threshold segmentation on the HDR panoramic grayscale image to obtain an HDR panoramic binary image;

[0081] An opening operation is performed on the HDR panoramic binary image to obtain a clean HDR panoramic binary image.

[0082] Optionally, the panoramic image projection format includes: any one of ERP format, EAP format, and CMP format.

[0083] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions, which, when executed, can complete the method of the aforementioned embodiment.

[0084] The embodiments of the present application also provide a computer program product for storing a computer program, wherein the computer program is used to execute the method of the aforementioned embodiment.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0086] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. A method for drawing light and shadow of a virtual object based on the position of a light source, characterized in that: include: When drawing the light and shadow of virtual objects, obtain high dynamic range HDR panoramic images; Preprocessing the HDR panoramic image to obtain an HDR panoramic binary image; Initialize a specified number of centroid points of the clustering clusters, and determine the cluster where each pixel point is located according to the distance between each pixel point in the HDR panoramic binary image and each centroid point; after re-determining the centroid points of each cluster, return to the step of determining the cluster where each pixel point is located according to the distance between each pixel point in the HDR panoramic binary image and each centroid point, until a preset condition is met, then stop clustering, and determine the position coordinates of at least one centroid point obtained as the image coordinates of at least one light source corresponding to the virtual object in the real scene, wherein the specified number is greater than or equal to 1; According to the panoramic image projection format, the image coordinates of the at least one light source are converted into three-dimensional coordinates in the real scene respectively; The light and shadow of the virtual object are drawn according to the three-dimensional coordinates of the at least one light source.

2. The method according to claim 1, characterized in that After obtaining the image coordinates of at least one light source corresponding to the virtual object in the real scene, the method further includes: For any one of the at least one light source, determine the number of pixels of a preset pixel value contained in a classification cluster to which a centroid point corresponding to the light source belongs; If the number of pixels is less than a preset pixel threshold, the image coordinates of the light source are discarded.

3. The method according to claim 1, characterized in that Drawing the light and shadow of the virtual object according to the three-dimensional coordinates of the at least one light source includes: Determining the distance between the at least one light source and the virtual object according to the three-dimensional coordinates of the at least one light source; According to the distance corresponding to each of the at least one light source, a light and shadow weight of the corresponding light source is set, wherein the light and shadow weight is positively correlated with the distance; The light and shadow of the virtual object are rendered according to the light and shadow weights of the at least one light source.

4. The method according to any one of claims 1 to 3, characterized in that The preprocessing of the HDR panoramic image to obtain an HDR binary image includes: Converting the HDR panoramic image into an HDR panoramic grayscale image; Performing adaptive threshold segmentation on the HDR panoramic grayscale image to obtain an HDR panoramic binary image; An opening operation is performed on the HDR panoramic binary image to obtain a clean HDR panoramic binary image.

5. The method according to any one of claims 1 to 3, characterized in that The panoramic image projection format includes: any one of an equirectangular projection ERP format, a cylindrical equal area projection EAP format, and a cube map projection CMP format.

6. A device for drawing light and shadow of a virtual object based on the position of a light source, characterized in that: Including display, communication interface, memory, processor; The display is connected to the processor and configured to display an augmented reality (AR) image; The communication interface is connected to the processor and is configured to send and receive data; The memory is connected to the processor and configured to store computer program instructions; The processor is configured to perform the following operations according to the computer program instructions: When drawing the light and shadow of virtual objects, obtain high dynamic range HDR panoramic images; Preprocessing the HDR panoramic image to obtain an HDR panoramic binary image; Initialize a specified number of centroid points of the clustering clusters, and determine the cluster where each pixel point is located according to the distance between each pixel point in the HDR panoramic binary image and each centroid point; after re-determining the centroid points of each cluster, return to the step of determining the cluster where each pixel point is located according to the distance between each pixel point in the HDR panoramic binary image and each centroid point, until a preset condition is met, then stop clustering, and determine the position coordinates of at least one centroid point obtained as the image coordinates of at least one light source corresponding to the virtual object in the real scene, wherein the specified number is greater than or equal to 1; According to the panoramic image projection format, the image coordinates of the at least one light source are converted into three-dimensional coordinates in the real scene respectively; The light and shadow of the virtual object are drawn according to the three-dimensional coordinates of the at least one light source.

7. The device according to claim 6, characterized in that The processor is further configured to: For any one of the at least one light source, determine the number of pixels of a preset pixel value contained in a classification cluster to which a centroid point corresponding to the light source belongs; If the number of pixels is less than a preset pixel threshold, the image coordinates of the light source are discarded.

8. The device according to claim 6, characterized in that The processor is specifically configured to: Determining the distance between the at least one light source and the virtual object according to the three-dimensional coordinates of the at least one light source; According to the distance corresponding to each of the at least one light source, a light and shadow weight of the corresponding light source is set, wherein the light and shadow weight is positively correlated with the distance; The light and shadow of the virtual object are rendered according to the light and shadow weights of the at least one light source.

9. The device according to any one of claims 6 to 8, characterized in that The processor is specifically configured to: Converting the HDR panoramic image into an HDR panoramic grayscale image; Performing adaptive threshold segmentation on the HDR panoramic grayscale image to obtain an HDR panoramic binary image; An opening operation is performed on the HDR panoramic binary image to obtain a clean HDR panoramic binary image.

10. The device according to any one of claims 6 to 8, characterized in that The panoramic image projection format includes any one of an equirectangular projection ERP format, a cylindrical equal area projection EAP format, and a cube map projection CMP format.