Information processing system and recording medium

Abstract

To provide a program for realizing an information processing apparatus.SOLUTION: An information processing apparatus 50 comprises communication means (e.g., 164) and imaging means 168. The program causes a computer to function as: first acquisition means 222 for acquiring a wide-angle image from an external imaging apparatus 10 via the communication means 164; calling means 226 for calling an imaging function by the imaging means 168; second acquisition means 228 for acquiring a captured image captured by the imaging means 168; and management means 230 for managing the wide-angle image acquired by the first acquisition means 222 and the captured image acquired by the second acquisition means 228 in association with each other.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] This disclosure relates to a program, a recording medium, and an information processing apparatus.

Background Art

[0002] Conventionally, in industries such as tourism, real estate, used cars, and weddings, still images and videos have been used for introducing and promoting products or facilities. In recent years, in addition to still images taken with a general camera, wide-angle images such as omnidirectional images that can convey a greater sense of presence have begun to be used. And, content is created by combining wide-angle images with still images and text information, and is utilized for the above-described purposes. As a method for shooting a wide-angle image for content creation, a method of using a combination of an omnidirectional camera and an application on a terminal such as a smartphone is known.

[0003] Regarding the creation of content using wide-angle images, the technique disclosed in Non-Patent Document 1 is known. The non-patent document discloses a method of performing an operation of setting a hot spot for scene movement at an arbitrary location in a wide-angle image while shooting, for the purpose of more efficiently generating content.

[0004] However, in a well-known method, when attempting to create content by associating a still image with a wide-angle image, it is necessary to copy an image taken with another digital camera to the terminal and associate it with the wide-angle image from the image folder on the terminal. Therefore, there is a need for time-consuming steps such as shooting in advance with another digital camera and then copying it to the terminal, and there is a possibility of associating an incorrect image from the image folder, leaving room for improvement.

Summary of the Invention

Problems to be Solved by the Invention

[0005] This disclosure is made in view of the above-mentioned prior art, and aims to provide a program that enables a computer equipped with imaging means to integrate the function of controlling an external imaging device to capture images and the function of capturing images with the imaging means provided by the computer, thereby enabling the centralized management of multiple types of captured images. [Means for solving the problem]

[0006] This disclosure provides a computer-executable program having the following features, which includes communication means and imaging means, in order to solve the above problems. The program causes the computer to function as a first acquisition means for acquiring a wide-angle image from an external imaging device via the communication means. The program also causes the computer to function as a calling means for calling the imaging function of the imaging means and as a second acquisition means for acquiring an image captured by the imaging means. Furthermore, the program causes the computer to function as a management means for associating and managing the wide-angle image acquired by the first acquisition means and the image captured by the second acquisition means. [Effects of the Invention]

[0007] With the above configuration, a computer equipped with an imaging means can integrate the function of controlling an external imaging device to capture images and the function of capturing images with the imaging means of the computer, making it possible to manage multiple types of captured images all at once. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram showing a 360-degree image content creation system including an information terminal according to this embodiment. [Figure 2] Figure 2 is a cross-sectional view of the 360-degree imaging device according to this embodiment. [Figure 3] Figure 3 shows the hardware configuration diagram of (A) a 360-degree imaging device and (B) an information terminal according to this embodiment. [Figure 4]Figure 4 is a functional block diagram relating to the function for creating 360-degree image content implemented on an information terminal according to this embodiment. [Figure 5] Figure 5 is a flowchart (1 / 2) showing the main process for creating 360-degree image content executed by the information terminal according to this embodiment. [Figure 6] Figure 6 is a flowchart showing the main process for creating 360-degree image content executed by the information terminal according to this embodiment (2 / 2). [Figure 7] Figure 7 is a flowchart showing the process for registering a floor plan that is performed by the information terminal according to this embodiment. [Figure 8] Figure 8 is a diagram illustrating how to associate locations on a floor plan. [Figure 9] Figure 9 is a flowchart showing the process for registering a 360-degree image performed by the information terminal according to this embodiment. [Figure 10] Figure 10 illustrates how the location on the 360-degree image is associated with the data. [Figure 11] Figure 11 is a flowchart showing the process for registering a normal image performed by the information terminal according to this embodiment. [Figure 12] Figure 12 shows the data structure for managing the 360-degree image content generated by the information terminal according to this embodiment. [Figure 13] Figure 13 shows the file structure when the 360-degree image content generated by the information terminal according to this embodiment is exported. [Figure 14] Figure 14 is an example of a screen displayed by the information terminal according to this embodiment. [Figure 15] Figure 15 is an example of a screen displayed by the information terminal according to this embodiment. [Figure 16] Figure 16 is an example of a screen displayed by the information terminal according to this embodiment. [Figure 17] Figure 17 is an example of a screen displayed by the information terminal according to this embodiment. [Figure 18]Figure 18 is an example of a screen displayed by the information terminal according to this embodiment. [Modes for carrying out the invention]

[0009] The embodiments of the present invention will be described below, but the embodiments of the present invention are not limited to those described below. In the embodiments described, an information processing device and a computer for realizing the information processing device will be described as an information terminal that communicates with a 360-degree imaging device and an application program installed on the information terminal, respectively, as examples.

[0010] Figure 1 is a diagram illustrating the schematic configuration of a 360-degree image content creation system 1 including an information terminal 50 according to this embodiment. The 360-degree image content creation system 1 shown in Figure 1 comprises a 360-degree imaging device 10 for capturing 360-degree images and an information terminal 50 capable of communicating with the 360-degree imaging device 10.

[0011] The information terminal 50 is not particularly limited, but a portable information terminal such as a smartphone, tablet computer, or laptop computer is preferred. An application program is installed on the information terminal 50 that controls the 360-degree imaging device 10 to capture 360-degree images and creates content (hereinafter referred to as 360-degree image content) using the captured 360-degree images. The information terminal 50 constitutes the information processing device or computer in this embodiment, equipped with a 360-degree image content creation function. Furthermore, the information terminal 50 and the 360-degree imaging device 10 can be combined to constitute the 360-degree image content creation system 1.

[0012] In the embodiment shown in FIG. 1, the omnidirectional imaging device 10 and the information terminal 50 are not particularly limited, but are connected by a wireless connection such as a wireless LAN (Local Area Network) or Bluetooth (registered trademark). An image in a predetermined format captured by the omnidirectional imaging device 10 is transmitted to the information terminal 50 via wireless communication, and omnidirectional image content is created using the image in the omnidirectional image content creation function. Note that the above connection form is an example, and a wired connection may be made via a wired LAN or USB (Universal Serial Bus).

[0013] In the embodiment shown in FIG. 1, further, the information terminal 50 is connected to the Internet 60 via a mobile communication network such as LTE (Long-Term Evolution) or 5G, or via communication devices such as an access point, a mobile router, or a broadband router.

[0014] In the embodiment shown in FIG. 1, a content publishing server 70 is provided on the Internet 60. The content publishing server 70 is a server that receives the omnidirectional image content sent from the information terminal 50 and stores and manages the received omnidirectional image content. The content publishing server 70 also responds to a request to view omnidirectional image content from the information terminal 50 or other information terminals, and transmits the requested omnidirectional image content to the requesting device. As a result, the omnidirectional image content can be viewed on the display device of the requesting device. The content publishing server 70 may be configured as a computer in an on-premises environment or a cloud environment.

[0015] In a specific embodiment, the content publishing server 70 can be configured as a web (application) server. It receives a request related to a content registration request including the omnidirectional image content to be registered in accordance with the HTTP (HyperText Transfer Protocol), accumulates the omnidirectional image content in an appropriate storage area, and manages it, for example, in a database. The content publishing server 70 further receives a request related to an image viewing request for which the omnidirectional image content is specified, reads out the target omnidirectional image content, and responds with a response including the content. In the requesting device that receives the response, the received omnidirectional image content can be viewed on a display device by a web browser or a specific application. In this embodiment, the content or the omnidirectional image content is a collection of data serving as one or more materials. The materials of this content or omnidirectional image content may include omnidirectional images, normal images, omnidirectional videos, normal videos, text (characters), voices, and the like. Also, in the embodiment to be described, as an example of the content, omnidirectional content including at least an omnidirectional image is used for description, but content using a wide-angle image other than the omnidirectional image instead of the omnidirectional image may also be used.

[0016] Hereinafter, the configurations of the omnidirectional imaging device 10 and the information terminal 50 according to this embodiment will be described while referring to FIGS. 2 and 3. FIG. 2 is a cross-sectional view of the omnidirectional imaging device 10 according to this embodiment. The omnidirectional imaging device 10 shown in FIG. 2 includes an imaging body 12, a housing 14 that holds the imaging body 12 and components such as a controller and a battery, and an operation button 18 provided on the housing 14.

[0017] The image sensor 12 shown in Figure 2 comprises two imaging optical systems 20A and 20B, and two image sensors 22A and 22B. The image sensors 22A and 22B are CCD (Charge Coupled Device) sensors or CMOS (Complementary Metal Oxide Semiconductor) sensors, etc. The imaging optical system 20 is configured, for example, as a fisheye lens with 6 groups and 7 elements. In the embodiment shown in Figure 2, the fisheye lens has a total field of view greater than 180 degrees (=360 degrees / n; number of optical systems n=2), and preferably has a field of view of 190 degrees or more. The number of imaging optical systems 20 and image sensors 22 is not particularly limited and may be 3 or more.

[0018] The optical elements (lenses, prisms, filters, and aperture diaphragms) of the two imaging optical systems 20A and 20B are positioned relative to the image sensors 22A and 22B. The optical axes of the optical elements of the imaging optical systems 20A and 20B are positioned perpendicular to the center of the light-receiving area of ​​the corresponding image sensor 22, and the light-receiving area becomes the image plane of the corresponding fisheye lens.

[0019] In the embodiment shown in Figure 2, the imaging optical systems 20A and 20B are identical in specifications and are combined in opposite directions so that their respective optical axes align. The image sensors 22A and 22B convert the received light distribution into an image signal and sequentially output image frames to the image processing block on the controller.

[0020] As will be described in detail later, the images captured by image sensors 22A and 22B are combined to generate an image with a solid angle of 4π steradians (hereinafter referred to as "spherical image"). The spherical image captures all directions that can be seen from the shooting location. In the embodiment described, it is preferable to describe the generation of a spherical image, but it may also be a panoramic image capturing only the horizontal plane at 360 degrees, or an image capturing a part of the entire spherical or horizontal 360-degree panorama. Furthermore, the spherical image may be a still image or a video.

[0021] Figure 3(A) shows the hardware configuration of the 360-degree imaging device 10 according to this embodiment. The 360-degree imaging device 10 includes a CPU (Central Processing Unit) 112, a ROM (Read Only Memory) 114, an image processing block 116, a video compression block 118, a still image compression block 119, a DRAM (Dynamic Random Access Memory) 132 connected via a DRAM interface 120, and an acceleration sensor 136 connected via an external sensor interface 124.

[0022] The CPU 112 controls the operation of each part of the 360-degree imaging device 10 and the overall operation. The ROM 114 stores control programs and various parameters written in code that the CPU 112 can decipher. The image processing block 116 is connected to two image sensors 130A and 130B (image sensors 22A and 22B in Figure 2), and receives image signals from images captured by each. The image processing block 116 includes an ISP (Image Signal Processor) and performs shading correction, Bayer interpolation, white balance correction, gamma correction, etc. on the image signals input from image sensors 130A and 130B. The image processing block 116 further synthesizes multiple images acquired from image sensors 130A and 130B to generate the 360-degree image described above.

[0023] The video compression block 118 is a codec block that performs video compression and decompression such as MPEG (Moving Picture Experts Group)-4 AVC (Advanced Video Coding) / H.264. The video compression block 118 is used to generate video data for the generated 360-degree image. The still image compression block 119 is a codec block that performs still image compression and decompression such as JPEG (Joint Photographic Experts Group) and TIFF (Tagged Image File Format). The still image compression block 119 is used to generate still image data for the generated 360-degree image. The DRAM 132 provides a memory area for temporarily storing data when performing various signal processing and image processing. The acceleration sensor 136 detects the acceleration components of the three axes, and the detected acceleration components are used to detect the vertical direction and perform zenith correction on the 360-degree image. The system is not limited to the acceleration sensor 136, and in addition to the acceleration sensor, it may also have the functions of a three-axis angular velocity sensor and an electronic compass, or both.

[0024] The 360-degree imaging device 10 further includes an external storage interface 122, a USB (Universal Serial Bus) interface 126, a serial block 128, and a video output interface 129. External storage 134 is connected to the external storage interface 122. The external storage interface 122 controls reading and writing to the external storage 134, such as a memory card inserted into the memory card slot.

[0025] A USB connector 138 is connected to the USB interface 126. The USB interface 126 controls USB communication with external devices such as the information terminal 50 and personal computers, which are connected via the USB connector 138. The serial block 128 controls serial communication with external devices such as the information terminal 50 and personal computers, and a wireless module 140 is connected to it. The video output interface 129 is an interface for connecting to an external display such as HDMI (High-Definition Multimedia Interface, HDMI is a registered trademark), and can output captured images to an external display or other device.

[0026] When the power is turned on by operating the power switch, the control program described above is loaded into the main memory. The CPU 112 controls the operation of each part of the device according to the program loaded into the main memory, and temporarily stores the data necessary for control in memory. This enables the various functions and processes of the 360-degree imaging device 10, which will be described later.

[0027] Figure 3(B) shows the hardware configuration of the information terminal 50 according to this embodiment. The information terminal 50 shown in Figure 3(B) includes a CPU 152, RAM 154, internal storage 156, input device 158, external storage 160, display 162, wireless module 164, USB connector 166, and built-in camera 168.

[0028] The CPU 152 controls the operation of each part of the information terminal 50 and the overall operation. The RAM 154 provides the CPU 152 with a workspace. The built-in storage 156 stores various data such as operating system programs and application programs that handle processing on the information terminal 50 side according to this embodiment, as well as images, all written in code that the CPU 152 can decipher.

[0029] The input device 158 is an input device such as a touchscreen and constitutes the input means in this embodiment, providing a user interface. The external storage 160 is a removable recording medium mounted in a memory card slot or the like, and records various types of data such as video image data and still image data. The wireless module 164 establishes a wireless LAN communication connection with external devices such as the 360-degree imaging device 10. The USB connector 166 establishes a USB connection with external devices.

[0030] In the following description, the wireless module 164 is used as the means of communication for communicating with the 360-degree imaging device 10. However, it is not limited to a specific standard, and external devices may be connected using other wireless communication methods such as Bluetooth® or wireless USB, or wired communication methods such as a USB connector 166 or a wired LAN (Local Area Network).

[0031] The display 162 displays an operation screen for user operation, displays a monitor image of the image captured by the 360-degree imaging device 10 before or during shooting, and displays saved videos and still images for playback and viewing. The display 162 and input device 158 allow the user to issue shooting instructions and change various settings to the 360-degree imaging device 10 via the operation screen. The display 162 constitutes the built-in display means in this embodiment. In addition, the information terminal 50 may be provided with a video interface such as HDMI®, and the screen can be output to an external display means such as a display device or projector device via the video interface.

[0032] The built-in camera 168 is a standard camera provided by the information terminal 50, and it captures normal still images or videos and generates still image data and video data. The built-in camera 168 constitutes the imaging means in this embodiment.

[0033] When power is supplied to the information terminal 50 and it enters the power-on state, various programs are read from the ROM and internal storage 156 and loaded into the RAM 154. The CPU 152 controls the operation of each part of the device according to the programs loaded into the RAM 154 and temporarily stores the data necessary for control in memory. This enables the various functions and processes of the information terminal 50, which will be described later. The above application program is the program in this embodiment for operating the computer as a means. When the above program is installed on the computer and the computer executes the program, the information terminal 50 according to this embodiment is realized.

[0034] While a detailed explanation of the hardware configuration of the content publishing server 70 is omitted here, it has a similar configuration to the information terminal 50 shown in Figure 3(B), and hardware components may be added, removed, or modified as appropriate depending on the application.

[0035] The function for creating 360-degree image content according to this embodiment will be described in more detail below with reference to Figure 4. Figure 4 is a functional block diagram related to the creation of 360-degree image content implemented on the information terminal 50 according to this embodiment.

[0036] As shown in Figure 4, the functional block 200 of the information terminal 50 includes a communication unit 202, a built-in camera control unit 204, a display control unit 206, an input unit 208, a 360-degree image content creation unit 220, and a content storage unit 240.

[0037] The communication unit 202 is a block that controls the wireless module 164, which is an interface with external devices, to perform communication with the 360-degree imaging device 10. The built-in camera control unit 204 is a block that controls the built-in camera 168, which is the imaging means described above.

[0038] The display control unit 206 is a block that controls the display 162, which is a display means. The input unit 208 is a block that controls the input device 158, which is an input means.

[0039] In a preferred embodiment, the 360-degree image content creation unit 220 integrates the function of controlling the 360-degree imaging device 10 to capture 360-degree images and the imaging function of the built-in camera 168 of the information terminal 50, and is provided as an application that centrally manages multiple types of images (360-degree images and normal images) captured by these devices.

[0040] The 360-degree image content creation unit 220 includes a 360-degree imaging device control unit 222, a built-in imaging function call unit 226, an image information management unit 230, and a UI (user interface) unit 234.

[0041] The 360-degree imaging device control unit 222 controls the external 360-degree imaging device 10 via the communication unit 202. The 360-degree imaging device control unit 222 issues commands to the 360-degree imaging device 10 to capture a 360-degree image and transmit the captured 360-degree image to the information terminal 50. The 360-degree imaging device control unit 222 also acquires a 360-degree image from the 360-degree imaging device 10 via the communication unit 202. The 360-degree imaging device control unit 222 constitutes the external device control means and the first acquisition means in this embodiment. The 360-degree image may be a still image or a video, but the following description will focus on the case where it is a still image.

[0042] The built-in imaging function call unit 226 uses the built-in camera control unit 204 to call the imaging function of the built-in camera 168. The built-in imaging function call unit 226 also acquires the image captured by the built-in camera 168 from the built-in camera control unit 204. The built-in imaging function call unit 226 constitutes the call means and the second acquisition means in this embodiment.

[0043] There are, for example, three implementation methods for directly or indirectly controlling the built-in camera 168 from an application. The first method is to call the camera application itself for taking images using the built-in camera 168. The second method is to call the camera function for taking images using the built-in camera 168. The third method is to implement the function for controlling the built-in camera 168 itself within the 360-degree image content creation unit 220. From the viewpoint of user operability, the second and third methods are preferred, but all implementation methods have in common that they allow the application to directly or indirectly control the built-in camera 168.

[0044] Furthermore, Figure 4 is compatible with any of the methods described above, in which the built-in imaging function call unit 226 calls the built-in camera control unit 204 as a camera function and calls the imaging function of the built-in camera 168. However, it is not limited to this.

[0045] In the embodiments described, it is assumed that the built-in camera 168 captures a normal still image (also referred to as a still image), but it may also capture a video. Furthermore, in the embodiments described, if a 360-degree image is present in the image folder (camera roll), the built-in camera 168 can be used to capture a normal image.

[0046] The image information management unit 230 manages the 360-degree spherical images acquired by the 360-degree imaging device control unit 222 and the normal images acquired by the built-in imaging function call unit 226 in association with each other, and stores the 360-degree spherical image content in the content storage unit 240. The 360-degree spherical image content consists of one or more 360-degree spherical images and one or more captured images. The image information management unit 230 constitutes the management means in this embodiment. The image information management unit 230 may also be configured to transmit (so-called upload) the created 360-degree spherical image content to an external content publication server 70 for registration, in addition to or instead of saving it to the content storage unit 240.

[0047] The image information management unit 230 shown in Figure 4 includes an association location identification unit 232. The association location identification unit 232 can identify a predetermined location in the 360-degree spherical image captured by the 360-degree imaging device 10 as an association destination to which the above-mentioned normal image or the like should be associated. The association location identification unit 232 constitutes the location identification means in this embodiment.

[0048] In a particular embodiment, the association position identification unit 232 can identify the central region of the display area when the 360-degree image is displayed on the display 162 as a predetermined position in the 360-degree image. Alternatively, in another embodiment, the association position identification unit 232 can identify a designated region specified by the input device 158 within the display area when the 360-degree image is displayed on the display 162 as a predetermined position in the 360-degree image.

[0049] Furthermore, the image information management unit 230 can associate the above-mentioned normal images with predetermined positions in the 360-degree image, as well as with content, which is a higher-level unit that includes the 360-degree image. Although not shown in Figure 4, the image information management unit 230 can manage both or either of the following in association with the 360-degree image or content: normal images acquired from the built-in imaging function call unit 226, images recorded in the built-in storage 156 or external storage 160 of the information terminal 50, and externally acquired images received from an image management system (e.g., content publishing server 70 or other services) with which the information terminal 50 can communicate. In addition, the image information management unit 230 can also manage the second 360-degree image by associating it with predetermined positions in the first 360-degree image, and in that case, it can also manage the first 360-degree image by associating it with predetermined positions in the second 360-degree image.

[0050] The UI unit 234 manages screen transitions related to the creation of 360-degree image content and provides a GUI (Graphical User Interface) including screens and buttons that constitute them. The UI unit 234 constitutes the first and second providing means in this embodiment, providing a first instruction unit that receives instructions to cause the 360-degree imaging device 10 to perform imaging, and a second instruction unit that receives instructions to call up the imaging function of the built-in camera 168, as GUI components.

[0051] The process for creating 360-degree image content in this embodiment will be described in more detail below with reference to Figures 5 to 18. Figures 5, 6, 7, 9, and 11 are flowcharts showing the process for creating 360-degree image content executed by the information terminal 50 in this embodiment. Note that Figures 5, 6, 7, 9, and 11 show the main processes related to associating multiple types of images in this embodiment, and other functions and exception handling processes are omitted. Figures 14 to 18 are diagrams illustrating various screens displayed on the display 162 of the information terminal 50 in this embodiment.

[0052] Figure 14(A) illustrates the top screen 400 of a 360-degree image content creation application. The top screen 400 shown in Figure 14(A) has a "Create Content" button 402 for starting the creation of 360-degree image content. The processes shown in Figures 5 and 6 start from step S100 shown in Figure 5 in response to the user tapping the "Create Content" button 402 on the top screen 400 shown in Figure 14(A).

[0053] In step S101, the information terminal 50 displays the content information input screen 410 shown in Figure 14(B). On the content information input screen 410 shown in Figure 14(B), the user can input information about the content, such as the content name and any alphanumeric characters that represent the content, using text boxes 412, 414, etc. The content information input screen 410 also has a "Map Insertion" button 416 for inserting a floor plan. By tapping the "Map Insertion" button 416, the user can invoke the process of inserting a predetermined floor plan image into the content. The content information input screen 410 also has an "OK" button 418 for completing the input of basic content information and proceeding to the next process of creating the content itself.

[0054] In step S102, the information terminal 50 branches the process according to the instructions received from the user on the content information input screen 410 shown in Figure 14(B). If in step S102 it is determined that the "Insert Map" button 416 has been tapped and an instruction to insert a floor plan has been received (insert floor plan), the process branches to step S103.

[0055] In step S103, the information terminal 50 displays a source selection menu that accepts the selection of a floor plan source, as shown in Figure 14(C). The source selection menu includes a "Take a picture with camera" button 422 for selecting the built-in camera 168 as the source, and a "Select an image from camera roll" button 424 for selecting an image folder within the information terminal 50. When either button is tapped, the process proceeds to step S104.

[0056] In the embodiment described, only the "Take a picture with camera" button 422 and the "Select an image from camera roll" button 424 are shown on screen 410 in Figure 14(C). However, buttons for selecting an external image management system on the network as a source may also be provided.

[0057] In step S104, the information terminal 50 performs a floor plan registration process. Here, the floor plan is the basic image that the application uses to associate a series of 360-degree images. In this embodiment, the floor plan (also called a floor map or layout plan) is used as the basic image that associates the entire content with each other. However, an image other than a floor plan may be used as the basic image, or there may be no floor plan, and the entire content may consist of two types of images: a 360-degree image and a normal image.

[0058] Figure 7 is a flowchart of the floor plan registration process. The process shown in Figure 7 is called in step S104 shown in Figure 5 and starts from step S201. In step S202, the information terminal 50 determines whether or not shooting with the built-in camera 168 has been selected. If, in step S202, the "Shoot with camera" button 422 has been tapped in the source selection menu and it is determined that shooting with the built-in camera 168 has been selected (YES), the process proceeds to step S203.

[0059] In step S203, the information terminal 50 calls the built-in camera control unit 204 using the built-in imaging function call unit 226. Figure 14(D) shows the shooting screen 430 for taking a picture of the floor plan with the built-in camera, which is displayed in response to the call to the built-in camera control unit 204. When the shooting button 432 is tapped on the shooting screen 430 shown in Figure 14(D) and the imaging process is completed, the process proceeds to step S204. In step S204, the information terminal 50 acquires a normal image captured by the built-in camera 168.

[0060] On the other hand, if it is determined in step S202 that shooting with the built-in camera is not selected (NO), the process proceeds to step S207. In step S207, the information terminal 50 determines whether or not it has been selected as the source. If it is determined in step S207 that the "Select image from camera roll" button 424 has been tapped in the source selection menu and the information terminal 50 has been selected (YES), the process branches to step S208.

[0061] In step S208, the information terminal 50 reads a list of images stored within the information terminal 50 (for example, in an image folder) and displays them in a list for selection. In step S209, the information terminal 50 accepts a selection from the image list within the information terminal 50, retrieves the selected image, and proceeds to step S205.

[0062] In step S205, the information terminal 50 determines whether an image has been acquired (acquired by photography, or acquired from internal or external sources). If it is determined in step S205 that an image has been acquired (YES), the process proceeds to step S206. In step S206, the information terminal 50 registers the acquired image as a floor plan associated with the content, records it as "Floor plan available", and returns to the process shown in Figure 5 in step S210. In this case, control returns to step S101, and the updated content information input screen is displayed.

[0063] Figure 15(A) shows the updated content information input screen 410 when a floor plan is set. In the content information input screen 410 shown in Figure 15(A), a new image of the floor plan 415 is displayed. In the embodiment described, the floor plan is an image captured by the built-in camera 168, but this is the case when taking a picture of a drawing on hand, and is not limited to a photograph; it may be an image such as a drawing or illustration that has been drawn in advance. This image may be saved in an image folder (camera roll), for example.

[0064] On the other hand, if it is determined that no image has been acquired (NO) in step S205, for example, if no image was taken or no image was selected, the process proceeds directly to step S210 and returns to the process shown in Figure 5. Also, if it is determined that no source has been selected (NO) in step S207, for example, if an area outside of buttons 422 and 424 is tapped in the source selection menu, the process proceeds directly to step S210 and returns to the process shown in Figure 5. Note that, in addition to buttons 422 and 424, the source selection menu may also be configured to allow acquisition of externally acquired images from an image management system (e.g., a content publishing server 70 or other services) with which the information terminal 50 can communicate.

[0065] Referring again to Figure 5, in step S102, if the "OK" button 418 is tapped on the content information input screen 410 shown in Figure 14(B) or Figure 15(A), and it is determined that an instruction to proceed to the next process for creating the content has been received (to the next screen), the process branches to step S105.

[0066] In step S105, the information terminal 50 branches the processing depending on whether or not a floor plan is available. If it is determined in step S105 that a floor plan is available (YES), the processing branches to step S106.

[0067] In step S106, the information terminal 50 displays the content creation screen (with floor plan) 440 in order to register a 360-degree image associated with a location on the floor plan. In step S107, the information terminal 50 branches the process according to the instructions received from the user on the content creation screen (with floor plan) 440. Note that when a floor plan has been registered and no 360-degree images have been registered yet, the content creation screen (with floor plan) 440 is in the state shown in Figure 15(B).

[0068] The content creation screen (with floor plan) 440 shown in Figure 15(B) includes a display area 442 where an image of the floor plan is displayed. This display area 442 is configured to acquire the coordinates of the position on the display area 442 where a position specification operation such as a tap is detected by the user. Optionally, the display area 442 may be configured to enlarge or reduce the image of the floor plan displayed within it in response to operations such as pinching.

[0069] Figure 8 illustrates how to associate a location with a floor plan. The display area 442 is displayed on the screen at a predetermined size, and the floor plan image is placed within the display area 442 at a predetermined scale. Therefore, the specified location 302 in the display area 442 can be converted from a relative position (X1 / (X1+X2), Y1 / (Y1+Y2)) as shown in Figure 8 to coordinates on the floor plan image. In the content creation screen (with floor plan) 440, when the display area 442 is tapped, the coordinate value on the floor plan image corresponding to the operated specified location in the display area 442 is identified, and an instruction to register a 360-degree image to that location is received. Note that in Figure 8, the top left is set as the origin and the coordinate values ​​are in the range of 0 to 1, but the way the origin is set is not limited, nor is the range of coordinate values ​​limited.

[0070] In the content creation screen (with floor plan) 440 shown in step S106, in addition to registering a 360-degree image ("Image Registration"), it is possible to access functions such as "Save" the content and "Confirm Image" to check the registered 360-degree images within the content. However, if no 360-degree images are registered at all, the GUI component for "Confirm Image" will not be displayed. On the other hand, the GUI component for "Save" the content may be displayed to allow temporary saving.

[0071] If it is determined in step S107 that "image registration" has been accepted on the content creation screen (with floor plan) 440, the process branches to step S108. In step S108, the information terminal 50 displays the source selection menu shown in Figure 15(C). The source selection menu includes a "shoot with 360-degree camera" button 452 for selecting the external 360-degree imaging device 10 as the source, and a "select 360-degree image from camera roll" button 454 for selecting an image folder within the information terminal 50. When either button is tapped, the process proceeds to step S109. The "shoot with 360-degree camera" button 452 constitutes the first instruction unit in this embodiment.

[0072] In step S109, the information terminal 50 sets the content and the position coordinate values ​​on the floor plan image corresponding to the specified location as the association destination for the 360-degree image to be registered, and in step S110, the information terminal 50 executes the 360-degree image registration process.

[0073] On the other hand, if it is determined in step S105 that there is no floor plan (NO), the process branches to step S111. In step S111, the information terminal 50 displays the content creation screen (without floor plan) 460. In step S112, the information terminal 50 branches the process according to the instructions received from the user on the content creation screen (without floor plan) 460. If no floor plan is registered and no 360-degree images have been registered yet, the content creation screen (without floor plan) 460 will be in the state shown in Figure 15(D). The content creation screen (without floor plan) 460 shown in Figure 15(D) also has a "Shoot with 360-degree camera" button 462 and a "Select 360-degree image from camera roll" button 464, similar to the screen shown in Figure 15(C). When either button is tapped, an instruction to register a 360-degree image ("Register Image") is received, and the process proceeds to step S112.

[0074] In the content creation screen (without floor plan) 460 shown in step S111, in addition to registering a 360-degree image ("Image Registration"), you can also access "Save" the content and check the registered 360-degree images within the content ("Image Confirmation"). When no 360-degree images are registered, the GUI component for "Image Confirmation" is not displayed, but the GUI component for "Save" the content may be displayed for temporary saving.

[0075] In step S112, if it is determined that "image registration" has been accepted on the content creation screen (without floor plan) 460, the process branches to step S113. In step S113, the information terminal 50 sets the content itself as the associated destination for the 360-degree image to be taken, and in step S110, the information terminal 50 executes the 360-degree image registration process.

[0076] Furthermore, in the embodiments described, only buttons 452, 454 and 462, 464 are shown in the screens of Figures 15(C) and 15(D), respectively. However, buttons for selecting captured but untransferred 360-degree images within the 360-degree imaging device 10, or buttons for selecting 360-degree images from an external image management system on the network, may also be provided as sources.

[0077] Figure 9 is a flowchart of the 360-degree image registration process. The process shown in Figure 9 is called in step S110 shown in Figure 5 and starts from step S300. In step S301, the information terminal 50 determines whether or not shooting with the external 360-degree imaging device 10 has been specified. If, in step S301, the source selection menu button 452 is tapped on screen 440 shown in Figure 15(C), or the button 462 is tapped on screen 460 shown in Figure 15(D), and it is determined that shooting with the 360-degree imaging device 10 has been selected (YES), the process proceeds to step S302.

[0078] In step S302, the information terminal 50 controls the 360-degree imaging device 10 via the 360-degree imaging device control unit 222 to capture a 360-degree image. Figure 16(A) shows the 360-degree shooting screen 470 for shooting with the 360-degree imaging device 10. When the shooting button 472 is tapped on the 360-degree shooting screen 470, the 360-degree imaging device control unit 222 issues an imaging command to the 360-degree imaging device 10 via the communication unit 202, the 360-degree imaging device 10 executes the imaging process, and once the imaging is complete, image transfer to the information terminal 50 begins. Figure 16(B) shows the transfer screen 480 indicating that the 360-degree image is being transferred from the 360-degree imaging device 10 to the information terminal 50. In step S303, the information terminal 50, via the 360-degree imaging device control unit 222, acquires the 360-degree image transferred from the 360-degree imaging device 10, which is an external imaging device, and proceeds to step S313.

[0079] On the other hand, if it is determined in step S301 that imaging with the 360-degree imaging device 10 is not selected (NO), the process branches to step S304. In step S304, the information terminal 50 determines whether or not an image already captured within the 360-degree imaging device 10, which is an external imaging device, is selected as the source. If it is determined in step S304 that an image already captured within the 360-degree imaging device 10 is selected as the source (YES), the process branches to step S305.

[0080] In step S305, the information terminal 50, via the 360-degree imaging device control unit 222, obtains a list of 360-degree images that have been captured (and not yet transferred to the information terminal 50) from the 360-degree imaging device 10 and displays them in a list for selection. In step S306, the information terminal 50 accepts the selection of a predetermined image from the list of 360-degree images in the 360-degree imaging device 10, receives the selected 360-degree image from the 360-degree imaging device 10 in step S303, and proceeds to step S313.

[0081] On the other hand, if it is determined in step S304 that no images already captured within the external 360-degree imaging device 10 have been selected (NO), the process branches to step S307. The determination in step S304 is performed if a button for selecting an already captured (and not yet transferred to the information terminal 50) 360-degree image from the 360-degree imaging device 10 as a source is provided in the menu shown in Figure 15(C) and the screen 460 shown in Figure 15(D). However, this step may be omitted, in which case the process may proceed directly to step S307.

[0082] In step S307, the information terminal 50 determines whether an image within the information terminal 50 has been selected as the source. If it is determined in step S307 that the information terminal 50 has been selected as the source (YES), the process branches to step S308.

[0083] In step S308, the information terminal 50 reads the list of 360-degree images stored within the information terminal 50. Note that the 360-degree images are in the same format as general images such as JPEG, but metadata is attached to identify whether or not they are 360-degree images. Based on this metadata, if an image that is not a 360-degree image is selected, the user can be prompted to select another image. Alternatively, the acquired list of images can be displayed separately from 360-degree images and other normally taken images, or only 360-degree images can be extracted in advance and included in the displayed list. It is also assumed that the information terminal 50 already has 360-degree images that have been taken and retrieved stored within it. In step S309, the information terminal 50 accepts a selection from the list of 360-degree images stored within the information terminal 50, reads the selected image, and proceeds to step S313.

[0084] On the other hand, if it is determined in step S307 that no selection has been made within the information terminal 50 (NO), the process branches to step S310. In step S310, the information terminal 50 obtains a list of 360-degree images via the network from an image management system (for example, a content publishing server 70) with which the information terminal 50 can communicate. It is assumed that the image management system already has 360-degree images that have been taken uploaded. In step S311, the information terminal 50 accepts a selection from the list of 360-degree images in the image management system, receives the 360-degree image via the network in step S312, and proceeds to step S313.

[0085] The processing in steps S310 to S312 is performed when a button for selecting an image management system on the network as the source is provided in the source selection menu shown in Figure 15(C) and screen 460 shown in Figure 15(D). However, this step may be omitted, in which case the process may proceed directly to step S316.

[0086] In step S313, the information terminal 50 displays a screen for confirming the 360-degree image to be used. In step S314, the information terminal 50 determines whether or not the 360-degree image to be used has been confirmed. Figure 16(C) shows the 360-degree image confirmation screen 490 for confirming the use of the 360-degree image. The 360-degree image confirmation screen 490 has a button 494 for confirming the use of the currently displayed 360-degree image and a button 492 for redoing the capture or selection.

[0087] In step S314, if button 494 is tapped on the 360-degree image confirmation screen 490 and it is determined that the image has been confirmed (YES), the process proceeds to step S315. In step S315, the information terminal 50 registers the confirmed 360-degree image as content associated with the association destination set in step S109 or S113 of Figure 5, and in step S316, the process returns to Figure 5. Note that this registration of the association destination can be moved to any position afterward. On the other hand, in step S314, if button 492 is tapped on the 360-degree image confirmation screen 490 shown in Figure 16(C) and it is determined that a redo is specified (NO), the process returns to step S301. Note that here, the explanation assumes that the redo is performed with the same source selected, but it is also possible to redo from the source selection.

[0088] Once the 360-degree image registration process shown in Figure 9 is complete, the process returns to step S105, and in step S106 or step S111, the updated content creation screen is displayed.

[0089] Figure 16(D) shows the content creation screen in the state where the floor plan has not been registered and one 360-degree spherical image has been registered, as updated in step S111.

[0090] The content creation screen (without floor plan) 460 shown in Figure 16(D) includes a display area 466 where the 360-degree image is displayed, and a "Save" button 468 for saving the content. The display area 466 also includes a GUI component 465 for calling up a screen to input the location where the 360-degree image was taken. The content creation screen (without floor plan) 460 also includes a "Select 360-degree image from camera roll" button 464 and an icon 469 for taking a picture with a 360-degree imaging device.

[0091] The display area 466 for a predetermined 360-degree image is a GUI component for calling up confirmation of registered 360-degree images ("Image Confirmation"). The "Save" button 468 is a GUI component for calling the content saving process. When one or more 360-degree images are registered, "Image Confirmation" is displayed, and the GUI component for "Save" content is displayed as needed.

[0092] Figure 18(D) shows the content creation screen when the floor plan has been registered and two 360-degree spherical images have been registered, as updated in step S106.

[0093] The content creation screen (with floor plan) 440 shown in Figure 18(D) is similar to the content creation screen (without floor plan) 460 shown in Figure 16(D), with the floor plan display area 442 displayed above the 360-degree image display area 446. Figure 18(D) shows a state where 360-degree images have been registered in association with two locations on the floor plan. Markers 442a and 442b are placed on the floor plan display area 442 at two locations. Figure 18(D) shows a state where the 360-degree image corresponding to marker 442b at the location indicated by "1" of the two markers 442a and 442b has been selected.

[0094] The display area 446 displays the selected 360-degree image and contains a GUI component 445 for calling up a screen to input the shooting location. The floor plan display area 442 also has a fan-shaped icon 442c superimposed on it, indicating the angle at which the 360-degree image is being displayed. The 360-degree image display area 446 is a GUI component for calling up confirmation of the registered and selected 360-degree image ("Image Confirmation"). The content creation screen (with floor plan) 440 shown in Figure 18(D) contains a "Save" button 448, similar to Figure 16(D).

[0095] The following describes an embodiment of adding annotations of a normal image to a 360-degree image using the content creation screen (without floor plan) 460 shown in Figure 16(D). However, as previously mentioned, the same process is carried out using the content creation screen (with floor plan) 440.

[0096] In the content creation screen (without floor plan) 460 shown in Figure 16(D), when the display area 466 corresponding to a predetermined 360-degree image is tapped, the instruction "Confirm Image" is received. If it is determined in step S107 or step S112 that "Confirm Image" has been received in the content creation screen 440 / 460, the process branches to step S114 shown in Figure 6 via point B shown in Figure 5.

[0097] In step S114, the information terminal 50 displays an image confirmation screen 500 for viewing a specified 360-degree image, as shown in Figure 17(A). The image confirmation screen 500 shown in Figure 17(A) includes a "back" button 502, a display area 504 for displaying the specified 360-degree image, and a menu button 508. This screen is configured to allow movement of the displayed portion of the 360-degree image displayed in the display area 504 in response to operations that indicate a direction, such as flicking.

[0098] In the image confirmation screen 500 shown in Figure 17(A), when the menu button 508 is tapped, a menu including the "Add Annotation" button 510 for performing image annotation is displayed on the image confirmation screen 500, as shown in Figure 17(B). When the "Add Annotation" button 510 is tapped, the instruction to add annotation is received. When the "Back" button 502 is tapped on the image confirmation screen 500 shown in Figure 17(A), the instruction to return to the content creation screen (for example, the content creation screen 460 in Figure 16(D)) is received.

[0099] If, in step S115, it is determined that the instruction to "add annotation" has been received on the image confirmation screen 500 shown in Figure 17(B), the process branches to step S116. In step S116, the information terminal 50 uses the association position identification unit 232 to identify the position to be associated. If "add annotation" is called via the menu button 508 on the image confirmation screen 500 shown in Figure 17(B), the position corresponding to approximately the center of the currently displayed portion in the display area 504 on the 360-degree image is identified as the association position to add the annotation.

[0100] Furthermore, in the above explanation, it was stated that the "Add Annotation" button 510 is displayed when the menu button 508 is tapped on the image confirmation screen 500 shown in Figure 17(A). However, the explanation is not limited to this, and the "Add Annotation" button 510 may also be displayed in response to a long tap (long press) of a predetermined position (e.g., 505) in the display area 504 on the image confirmation screen 500 shown in Figure 17(A). When adding an annotation is invoked via tapping an area in the display area 504, the position on the 360-degree image corresponding to the position when the area in the display area 504 was tapped is identified as the associated position for adding the annotation.

[0101] Figure 10 illustrates how the location on the 360-degree image is associated with the data structure. As shown in the data structure 330 of Figure 10, the 360-degree image is generated in a format where the horizontal axis represents longitude from -180 degrees to +180 degrees and the vertical axis represents latitude from -90 degrees to +90 degrees.

[0102] When displaying this 360-degree image on a flat device such as the display 162 of the information terminal 50, the 360-degree image can be attached to a spherical object 310, for example, by placing a virtual camera 312 at the center of the spherical object 310, and displayed as a projected image 314 observed from that virtual camera 312. In other words, the display area 504 is displayed on the display with a predetermined size, and the projected image 314 generated from the 360-degree image is placed within that display area 504. The position on the 360-degree image corresponding to the center position in the projected image 314 being displayed can be obtained from the orientation of the virtual camera 312. On the other hand, any position in the display area 504 (for example, a tapped position) 322 can be converted from a relative position (X1 / (X1+X2), Y1 / (Y1+Y2)) as shown in Figure 10, via a point 316 in the range corresponding to the projected image 314 on the spherical object 310, to a coordinate 332 on the 360-degree image format 330. In Figure 10, the center is set as the origin, and the coordinate values ​​are set within the range of longitude -180 degrees to +180 degrees and latitude -90 degrees to +90 degrees. However, the origin is not limited to any particular location, nor is the range limited. For example, it could be expressed as longitude from 0 degrees to 360 degrees, or latitude from 0 degrees to 180 degrees. Furthermore, if the wide-angle image is an image with a field of view of 180 degrees or more other than a 360-degree spherical image, the image coordinates will be set in a manner appropriate to that field of view, instead of the coordinates of the 360-degree spherical image shown above.

[0103] Referring again to Figure 6, in step S117, the information terminal 50 sets the identified location on the 360-degree image as the associated location. In step S118, the information terminal 50 displays the annotation setting screen 520 shown in Figure 17(C). The annotation setting screen 520 shown in Figure 17(C) has a "Select Image" button 522. By tapping the "Select Image" button 522, the user can invoke the process of registering an image captured by the built-in camera 168 of the information terminal 50. Although this example shows adding only an image as annotation, it is also possible to add text along with the image by providing a text input field.

[0104] In step S119, the information terminal 50 displays a source selection menu that accepts the selection of a source for a normal image, as shown in Figure 17(D). The source selection menu includes a "Take a picture with camera" button 524 for selecting the built-in camera as the source, and a "Select an image from camera roll" button 526 for selecting an image folder within the information terminal 50. When either button is tapped, the process proceeds to step S120. The "Take a picture with camera" button 524 constitutes the second instruction unit in this embodiment.

[0105] In step S120, the information terminal 50 performs the normal image registration process.

[0106] Figure 11 is a flowchart showing the process for registering a normal image. The process shown in Figure 11 is called in step S120 shown in Figure 5 and starts from step S400. In step S401, the information terminal 50 determines whether or not shooting with the built-in camera has been specified. If, in step S401, button 524 has been tapped in the source selection menu in Figure 17(D) and it is determined that shooting with the built-in camera 168 has been selected (YES), the process proceeds to step S402.

[0107] In step S402, the information terminal 50 calls the built-in camera control unit 204 using the built-in imaging function call unit 226. Figure 18(A) shows the shooting screen 530 for taking a normal image with the built-in camera, which is displayed in response to the call to the built-in camera control unit 204. When the shooting button 532 is tapped on the shooting screen 530 shown in Figure 18(A) and the imaging process is completed, the process proceeds to step S410. In step S403, the information terminal 50 acquires the normal image captured by the built-in camera 168 and the process proceeds to step S410.

[0108] On the other hand, if it is determined in step S401 that shooting with the built-in camera 168 is not selected (NO), the process branches to step S404. In step S404, the information terminal 50 determines whether or not it is selected as a source. If it is determined in step S404 that the information terminal 50 is selected as a source (YES), the process branches to step S405.

[0109] In step S405, the information terminal 50 reads the image list within the information terminal 50 and displays it as a list for selection. In step S406, the information terminal 50 accepts a selection from the image list within the information terminal 50, reads the selected image, and proceeds to step S410.

[0110] On the other hand, if it is determined in step S404 that no selection has been made within the information terminal 50 (NO), the process branches to step S407. In step S407, the information terminal 50 obtains a list of regular images via the network from an image management system (e.g., a content publishing server 70) with which the information terminal 50 can communicate, and displays the list. In step S408, the information terminal 50 accepts a selection from the list of regular images within the image management system, receives a regular image via the network in step S409, and proceeds to step S410. Note that the processing in steps S407 to S409 is performed if a button for selecting the image management system as the source is provided in the menu shown in Figure 17(D), but this may be omitted, in which case the process may proceed directly to step S413.

[0111] In step S410, the information terminal 50 displays a screen for confirming the normal image to be used. In step S410, the information terminal 50 determines whether or not the normal image to be used has been confirmed. Figure 18(B) shows the image confirmation screen 540 for confirming the normal image. The image confirmation screen 540 has a button 542 for confirming the use of the currently displayed image and a button 544 for redoing the shooting or selection.

[0112] If, in step S411, button 542 is tapped on the image confirmation screen 540 and it is determined that the image has been confirmed (YES), the process proceeds to step S412. In step S412, the information terminal 50 associates the confirmed normal image with the associated destination and registers the content, and in step S413, the process returns to Figure 6. The associated destination here is the coordinate value of a predetermined position on the 360-degree image identified in step S117, or the orientation indicating that predetermined position. Note that this registered associated destination can be moved to any position afterward. On the other hand, if, in step S411, button 544 is tapped on the image confirmation screen 540 and it is determined that a redo is specified (NO), the process returns to step S401.

[0113] Returning to the process in Figure 6, the process is returned in step S114 via point B. In step S114, the information terminal 50 displays the updated image confirmation screen 500 as shown in Figure 18(C). In the image confirmation screen 500 shown in Figure 18(C), a marker 550 indicating the position to which the normal image is associated is displayed in the display area 504 of the 360-degree image. The coordinates of this marker 550 correspond to the coordinates set as the association destination when the normal image was associated and registered in step S412. Also, since there is one or more annotations, a button 552 to delete the annotation and a button 554 to edit the annotation are displayed, but we will not go into further detail about deleting and editing annotations.

[0114] In step S115, if the information terminal 50 determines that the "Back" 502 has been tapped on the image confirmation screen 500 shown in Figure 17(A) or Figure 18(C), and that it has received an instruction to return to a higher level, it branches the process to step S105 shown in Figure 5 via point C. In this case, further processing is performed to register another 360-degree image associated with a predetermined location on the floor plan of the content or with the content itself.

[0115] At this point, we assume that either a content creation screen (without floor plan) 460, as shown in Figure 16(D), or a content creation screen (with floor plan) 440, in which a 360-degree image is registered in a predetermined position on the floor plan, is displayed. If it is determined in step S107 or step S112 that a "Save" instruction has been received on the content creation screen 440 / 460, the process branches to step S121 shown in Figure 6 via point A.

[0116] In step S121, the information terminal 50 saves the created 360-degree image content to the content storage unit 240, and in step S122, the process is terminated. Alternatively, instead of saving to the content storage unit 240, or in conjunction with saving to the content storage unit 240, the 360-degree image content may be uploaded to the content publishing server 70.

[0117] While I won't go into further detail, the system may include functions to delete and replace registered 360-degree images and regular images, to apply image processing such as blurring, stamping, and resizing to designated areas of 360-degree images and regular images, and to input, change, and delete names of content and 360-degree images before saving and uploading the 360-degree image content.

[0118] In the explanation above, a regular image was described as being associated with a predetermined location within a 360-degree image as an annotation. However, this is not limited to regular images; other 360-degree images may be added, and other types of information such as text or audio can also be added as annotations. Furthermore, while a regular image was described as being associated with a predetermined location within a 360-degree image, it may also be associated with the entire 360-degree image, or with the content, which is a higher-level unit that includes the 360-degree image.

[0119] For example, in content showcasing real estate properties and their facilities, regular images of the facilities in each room may be associated with 360-degree images of each room at predetermined positions, while regular images of the entrance or view from the balcony may also be taken. In this case, the regular images can be associated with the entire content rather than with the 360-degree images of each room. Alternatively, the 360-degree images of each room may be associated with each other at predetermined positions (for example, the position corresponding to a door).

[0120] Figure 12 shows the data structure for managing the 360-degree image content generated by the information terminal 50 according to this embodiment. Figure 12(A) shows a content information management table that manages information such as content identifiers (IDs) and names in association. Figure 12(B) shows a 360-degree image management table that manages 360-degree image IDs and their storage locations in association. Figure 12(C) shows a normal image management table that manages normal image IDs and their storage locations in association.

[0121] Figure 12(D) shows a content element management table that manages the correspondence between the ID of a content element to be associated with a 360-degree image, such as a room ID, its name, the associated 360-degree image ID, and its coordinates on the floor plan. Here, the coordinates on the floor plan are obtained and stored as explained earlier, for example in Figure 8. In the table, Null indicates that no value exists. In other words, content ID=102 indicates content that does not include a floor plan or a room (content element) that is not associated with a location on the floor plan.

[0122] Figure 12(E) shows a floor plan management table that manages floor plan IDs, content IDs, and normal image IDs that identify the normal image corresponding to the floor plan. Figure 12(F) shows an annotation management table that manages annotation IDs, 360-degree image IDs associated with the annotation, coordinates on the 360-degree image where the annotation is placed, a description of the annotation, and a normal image ID that identifies the normal image to be added as the annotation. Here, the coordinates on the 360-degree image are obtained as explained in Figure 10. Note that in the annotation management table, instead of associating with the 360-degree image, the table configuration could be such that the entire content is associated.

[0123] Figure 13 shows the file structure when the 360-degree image content generated by the information terminal 50 according to this embodiment is exported. In Figure 13, under the folder named "Content," there are folders named "Floor Plan Image File," "View Image File," and "360-degree Image File." Each folder named "360-degree Image" contains the 360-degree image file and a regular image file associated with the 360-degree image.

[0124] The process described above creates a 360-degree image content having a data structure schematically shown in Figure 12, in which a regular image is associated with an arbitrary position in the 360-degree image. When viewed in a viewer, the 360-degree image content can be displayed with the regular image embedded within the 360-degree image as an annotation. For example, in the viewer, in response to a predetermined operation (e.g., tap, click) on an image element (marker, icon, symbol) placed at an annotation position in the 360-degree image, the regular image associated with that position is displayed as a pop-up.

[0125] As described above, according to the embodiment described above, it is possible to provide a program for realizing an information processing device (computer) equipped with imaging means that integrates the function of controlling an external imaging device to take images and the function of taking images with the imaging means provided by the information processing device (computer), and that can manage multiple types of images taken at once, a recording medium on which the program is recorded, and the information processing device.

[0126] In the embodiment described above, the function of controlling an external imaging device to take images and the function of taking images using the imaging means provided by the information processing device itself are integrated into a single application. Therefore, the user does not need to use the function of controlling an external imaging device to take images and the function of taking images using the imaging means in separate applications.

[0127] Furthermore, when using individual imaging functions in different applications, the images are first saved to an image folder such as the camera roll, and then registered to the content via the image folder. However, in this case, although images may be saved chronologically within the image folder, images that should be registered to different content may be mixed together, making it difficult to correctly register the images to the content. In contrast, in the embodiment described above, multiple types of images can be managed together, thus reducing the confusion that occurs when using image folders.

[0128] Furthermore, by combining the capture of 360-degree images and regular images for content creation, the effort required of the creator can be reduced. Since the creator can capture regular images using the content creation application installed on the information terminal 50, if the 360-degree imaging device 10 and the information terminal 50 are available, regular cameras and other shooting equipment become unnecessary, eliminating the need for the conventional effort of capturing regular images and videos. In addition, linking content, floor plans (as needed), 360-degree images, and regular images can be done effortlessly, reducing the effort required for classifying and managing captured regular images.

[0129] In the embodiments described above, a 360-degree panoramic image content, including a 360-degree panoramic image as a wide-angle image, was described as an example of the content to be created. However, the wide-angle image is not limited to the 360-degree panoramic image described above, and may also be a panoramic image of 360 degrees horizontally, or an image that captures a part of a 360-degree panoramic view. Here, a wide-angle image is an image with a relatively large field of view compared to a normal image, and preferably an image with a field of view of at least 180 degrees horizontally. Furthermore, a wide-angle image may be an image with a large field of view captured by one or more imaging devices. Furthermore, a normal image may be a still image or a video, and is an image with a narrower field of view than a wide-angle image, more specifically, an image with a field of view of 100 to 25 degrees, preferably between 70 and 45 degrees.

[0130] Furthermore, in the embodiments described above, the 360-degree image and the normal image were described as still images. However, the 360-degree image and the normal image are not limited to still images, but may also be videos or semi-videos (time-lapse videos, interval-shot images, animations) composed of a series of still images. In that case, in addition to the position of the 360-degree image described above, the images are associated with each other by further linking them to temporal information such as the time and frame interval.

[0131] Each of the embodiments described above can be implemented by one or more processing circuits. Here, "processing circuit" in this disclosure includes processors programmed to execute each function by software, such as processors implemented by electronic circuits, and devices such as ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), FPGAs (field programmable gate arrays), and conventional circuit modules designed to execute each of the functions described above.

[0132] Furthermore, the above functions can be implemented by computer-executable programs written in legacy programming languages ​​such as assembler, C, C++, C#, Java (registered trademark), or object-oriented programming languages, and can be stored on device-readable recording media such as ROM, EEPROM, EPROM, flash memory, flexible disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disc, SD card, MO, etc., or distributed via telecommunication lines.

[0133] While embodiments of the present invention have been described above, the embodiments of the present invention are not limited to those described above. Other embodiments, additions, modifications, and deletions are possible within the scope that a person skilled in the art can conceive, and any embodiment that achieves the function and effect of the present invention is included within the scope of the present invention. [Explanation of Symbols]

[0134] 10...Spherical imaging device, 12...Imager, 14...Housing, 18...Operation buttons, 20...Imaging optical system, 22...Image sensor, 50...Information terminal, 60...Internet, 70...Content publishing server, 112...CPU, 114...ROM, 116...Image processing block, 118...Video compression block, 119...Still image compression block, 120...DRAM interface, 122...External storage interface, 124...External sensor interface, 126...USB interface, 128...Serial block, 129...Video output interface, 130...Image sensor, 132...DRAM, 134...External storage, 1 36…Accelerometer, 138…USB connector, 140…Wireless module, 152…CPU, 154…RAM, 156…HDD, 158…Input device, 160…External storage, 162…Display, 164…Wireless module, 166…USB connector, 168…Built-in camera, 200…Function block, 202…Communication unit, 204…Built-in camera control unit, 206…Display control unit, 208…Input unit, 220…Spherical image content creation unit, 222…Spherical imaging device control unit, 226…Built-in imaging function call unit, 230…Image information management unit, 232…Association location identification unit, 234…UI unit, 240…Content storage unit [Prior art documents] [Non-patent literature]

[0135] [Non-Patent Document 1] “How to capture a 360° tour”, [online], released on July 19, 2016, youtube, [searched on February 5, 2020], Internet <url: https: youtu.be ba3eamnu_cw?t="162">< / url:>

Claims

1. An information processing system including an information processing device equipped with a communication unit and an imaging unit, and an external imaging device for capturing wide-angle images, The aforementioned information processing device has an application program installed on it. The aforementioned application is A control screen for controlling the external imaging device via the communication unit is displayed on the information processing device, the control screen displays a wide-angle image captured by the external imaging device, the external imaging device is made to capture a wide-angle image in response to an operation on the control screen, and the captured wide-angle image is transmitted to the information processing device in response to the operation. The imaging unit has a function to control the acquisition of normal images, A function for associating the wide-angle image captured by the external imaging device with at least one of the normal images captured by the imaging unit, A function to create content based on the associated wide-angle image and at least one normal image, wherein when the content is displayed on a display device, a first display component indicating that the normal image is associated with the wide-angle image is displayed on the wide-angle image. An information processing system equipped with the following features.

2. The aforementioned application is The system includes a function to identify a predetermined position in the wide-angle image, The association function associates the normal image with the predetermined position identified in the wide-angle image. The information processing system according to claim 1.

3. The function described above creates the content such that when the content is displayed on the display device, the first display component is displayed at the specified predetermined position in the wide-angle image, and in response to an operation on the first display component, the normal image associated with the predetermined position is displayed. The information processing system according to claim 2.

4. The function for identifying the predetermined position identifies the center position of the display area when the wide-angle image is displayed on the display device as the predetermined position in the wide-angle image. The information processing system according to claim 2.

5. The aforementioned application is It has a function to accept the specification of a predetermined position in the wide-angle image, The association function identifies the area specified by the receiving function as the predetermined position in the wide-angle image within the display area when the wide-angle image is displayed on the display device. The information processing system according to claim 2.

6. The aforementioned application is The system includes a function to identify the imaging position of the wide-angle image in the base image to which the wide-angle image is associated, The association function associates the wide-angle image with the imaging position identified in the basic image. The information processing system according to claim 1.

7. The function described above creates the content such that when the content is displayed on the display device, a second display component indicating the imaging position is displayed on the basic image. The information processing system according to claim 6.

8. The function described above creates content such that, when the basic image is displayed on the display device, the wide-angle image associated with the imaging position is displayed in response to an operation on the second display component. The information processing system according to claim 7.

9. The association function manages both or either of the recorded images recorded on the recording medium of the information processing device and the externally acquired images received from an image management system with which the information processing device can communicate, by associating them with the wide-angle image. The information processing system according to claim 7.

10. A recording medium for storing a computer-readable program for realizing an information processing device in an information processing system including an information processing device having a communication unit and an imaging unit, and an external imaging device for capturing wide-angle images, wherein the program is an application program installed on the information processing device, The aforementioned application is A control screen for controlling the external imaging device via the communication unit is displayed on the information processing device, the control screen displays a wide-angle image captured by the external imaging device, the external imaging device is made to capture a wide-angle image in response to an operation on the control screen, and the captured wide-angle image is transmitted to the information processing device in response to the operation. The imaging unit has a function to control the acquisition of normal images, A function for associating the wide-angle image captured by the external imaging device with at least one of the normal images captured by the imaging unit, A function to create content based on the associated wide-angle image and at least one normal image, wherein when the content is displayed on a display device, a first display component indicating that the normal image is associated with the wide-angle image is displayed on the wide-angle image. A recording medium equipped with the following features.

Images