Information Processing Apparatus and Method, Storage Medium, and Computer Device

Abstract

The present invention relates to an information processing apparatus and method, a storage medium, and a computer apparatus, aiming to integrate the function of controlling the photography of an external photography apparatus and the function of photographing with a photography unit provided in the information processing apparatus, and uniformly manage various types of photographed images. The information processing apparatus includes a communication unit (164) and a photography unit (168), and has a first acquisition unit (222) for acquiring a wide-angle image from an external photography apparatus (10) through the communication unit (164); a call unit (226) for calling the photography function of the photography unit (168); a second acquisition unit (228) for acquiring a photographed image taken by the photography unit (168); and a management unit (230) for managing the wide-angle image acquired by the first acquisition unit (222) in association with the photographed image acquired by the second acquisition unit (228).

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and method, a storage medium, and a computer apparatus. Background Art

[0002] Conventionally, industries such as tourism, real estate, used cars, and weddings often use still images or animations to introduce and promote products and facilities. In addition to still images taken by ordinary cameras, in recent years, more immersive wide-angle images such as omnidirectional images have also been used. For the above-mentioned promotional purposes, wide-angle images are used in combination with still images and text information to create content. Regarding the photographic method of wide-angle images used to create content, a method of combining an omnidirectional camera and an application on a terminal such as a smartphone is known.

[0003] Regarding creating content using wide-angle images, Non-Patent Document 1 discloses a method of photographing while performing a hot spot setting operation of moving a scene to an arbitrary place in a wide-angle image for the purpose of effectively generating content.

[0004] Non-Patent Document 1: “How to capture a 360°tour”, online, published on July 19, 2016, youtobe, retrieved on February 5, Reiwa 2, Internet <URL: https: / / youtu.be / BA3EaMNU_Cw?t=162>

[0005] However, in currently well-known methods, when associating a wide-angle image with a still image to create content, it is necessary to copy an image taken by another digital camera to a terminal and associate it with the wide-angle image from an image folder on the terminal. For this purpose, not only is it necessary to photograph in advance with another digital camera, but also to copy it to the terminal, which is time-consuming and laborious, and furthermore, there is a possibility of associating with an incorrect image from the image folder, so there is still room for improvement. Summary of the Invention

[0006] In view of the above problems in the prior art, the present invention aims to provide an information processing apparatus having a photographing unit that can integrate a function for controlling photographing of an external imaging device and a photographing function of a photographing unit provided in the above information processing apparatus, and uniformly manage various types of captured images.

[0007] To achieve the above object, the present invention provides an information processing apparatus having a communication unit and a photographing unit, characterized by having a first acquisition unit for acquiring a wide-angle image from an external photographing device through the communication unit; a call unit for calling a photographing function of the photographing unit; a second acquisition unit for acquiring a photographed image taken by the photographing unit; and a management unit for associating and managing the wide-angle image acquired by the first acquisition unit and the photographed image acquired by the second acquisition unit.

[0008] The effect of the present invention is that it can integrate the function of controlling the photography of an external photography device in an information processing device equipped with a photographing unit and the photographing function of the photographing unit provided in the information processing device, and uniformly manage various types of images captured. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a schematic diagram of a full-sphere image content creation system having an information terminal according to this embodiment.

[0010] Figure 2 It is a sectional view of a full-sphere photography device according to this embodiment.

[0011] Figure 3 Among them, (A) is a hardware structure module diagram of the full-sphere photography device according to this embodiment, and (B) is a hardware structure module diagram of the information terminal according to this embodiment.

[0012] Figure 4 It is a functional structure module diagram of the full-sphere image content creation function in the information terminal according to this embodiment.

[0013] Figure 5 It is a main processing flow chart for creating full-sphere image content executed by the information terminal according to this embodiment.

[0014] Figure 6 It is a main processing flow chart for creating full-sphere image content executed by the information terminal according to this embodiment ( Figure 5 continued).

[0015] Figure 7 It is a flow chart of room type map login processing executed by the information terminal according to this embodiment.

[0016] Figure 8 It is a schematic diagram for explaining a method related to the position on the room type map.

[0017] Figure 9 It is a flow chart of full-sphere image login processing executed by the information terminal according to this embodiment.

[0018] Figure 10 It is a schematic diagram for explaining a method related to the position on the full-sphere portrait.

[0019] Figure 11 It is a flow chart of ordinary image login processing executed by the information terminal according to this embodiment.

[0020] Figure 12 It is a schematic diagram of the data structure for managing full-sphere image content generated by the information terminal according to this embodiment.

[0021] Figure 13 It is a schematic diagram of the file structure when the all-sky image content generated by the information terminal involved in this embodiment is output.

[0022] Figure 14 It is a schematic diagram of the screen displayed by the information terminal involved in this embodiment for illustration.

[0023] Figure 15 It is a schematic diagram of the screen displayed by the information terminal involved in this embodiment for illustration.

[0024] Figure 16 It is a schematic diagram of the screen displayed by the information terminal involved in this embodiment for illustration.

[0025] Figure 17 It is a schematic diagram of the screen displayed by the information terminal involved in this embodiment for illustration.

[0026] Figure 18 It is a schematic diagram of the screen displayed by the information terminal involved in this embodiment for illustration. Detailed implementation mode

[0027] The following describes the embodiments of the present invention, but the embodiments of the present invention are not limited by the following embodiments. In the following embodiments, an information terminal communicating with an all-sky photography device and an application program installed in the information terminal are taken as examples to illustrate an information processing device and a computer-executable program for implementing the information processing device.

[0028] Figure 1 It is a schematic diagram of the schematic configuration of the all-sky image content creation system 1 including the information terminal 50 involved in this embodiment. Figure 1 The shown all-sky image content creation system 1 includes an all-sky photography device 10 for photographing an all-sky image and an information terminal 50 that can communicate with the all-sky photography device 10.

[0029] There is no particular limitation on the information terminal 50, but a portable information terminal such as a smart phone, a tablet computer, or a notebook computer is preferred. An application program for controlling the all-sky photography device 10 to photograph an all-sky image and creating content (hereinafter referred to as all-sky image content) with the photographed all-sky image is installed on the information terminal 50. The information terminal 50 constitutes the information processing device or computer device of this embodiment having an all-sky image content creation function. The information terminal 50 and the all-sky photography device 10 together constitute the all-sky image content creation system 1.

[0030] In Figure 1In the illustrated embodiment, the omnidirectional imaging device 10 and the information terminal 50 are not particularly limited, and are wirelessly connected via wireless LAN (Local Area Network), Bluetooth (registered trademark), etc. The image in a specified format captured by the omnidirectional imaging device 10 is wirelessly communicated to the information terminal 50, and omnidirectional image content is created using this image by the omnidirectional image content creation function. The above connection method is an example, and a wired connection may also be made via a wired LAN or USB (Universal Serial Bus).

[0031] Furthermore, in Figure 1 the illustrated embodiment, the information terminal 50 is connected to the Internet 60 via a mobile communication network such as LTE (Long-Term Evolution) or 5G, or a communication device such as an access point, a mobile router, or a broadband router.

[0032] In Figure 1 the illustrated embodiment, a content disclosure server 70 on the Internet 60 is provided. The content disclosure server 70 is a server that receives the omnidirectional image content transmitted from the information terminal 50 and stores and manages the received omnidirectional image content. The content disclosure server 70 responds to a request for viewing omnidirectional image content from the information terminal 50 or another information terminal, and transmits the omnidirectional image content to the requesting device. In this way, the omnidirectional image content can be viewed on the display device of the requesting device. The content disclosure server 70 may also be configured by a computer in an on-premises environment or a cloud environment.

[0033] In a specific embodiment, the content disclosure server 70 may be a web (application) server. According to the HTTP (Hypertext Transfer Protocol), the content disclosure server 70 receives a content registration request that includes the omnidirectional image content to be registered, and stores the omnidirectional image content in a corresponding storage area, for example, manages it in a database. The content disclosure server 70 also receives a request for viewing an image specified by the omnidirectional image content, reads the omnidirectional image content as the object, and returns a response including the content. The requesting device that receives the response can view the received omnidirectional image content on the display device via a web browser or a specific application. In the present embodiment, the content or omnidirectional image content is a data set of one or more materials. The materials of the content or omnidirectional image content may include omnidirectional images, ordinary images, omnidirectional videos, ordinary videos, text (characters), voices, etc. In the embodiments described below, an omnidirectional content including at least an omnidirectional image is taken as an example for description, but it may also be content of a wide-angle image other than the omnidirectional image.

[0034] The following refers toFigure 2 And Figure 3 , which illustrates the configurations of the omnidirectional imaging device 10 and the information terminal 50 according to this embodiment. Figure 2 It is a sectional view of the omnidirectional imaging device 10 according to this embodiment. Figure 2 The illustrated omnidirectional imaging device 10 is provided with an imaging body 12, a housing 14 that holds components such as the above-mentioned imaging body 12, a controller, and a battery, and operation buttons 18 provided on the housing 14.

[0035] Figure 2 The illustrated imaging body 12 includes two imaging optical systems 20A, 20B and two imaging elements 22A, 22B. The imaging elements 22A, 22B are CCD (Charge Coupled Device) sensors, CMOS (Complementary Metal Oxide Semiconductor) sensors, etc. The imaging optical system 20 is composed of a six-group seven-element fisheye lens. In Figure 2 the illustrated embodiment, the above-mentioned fisheye lens has a full angle greater than 180 degrees (= 360 degrees / n; optical coefficient n = 2), and preferably has a viewing angle of 190 degrees or more. There is no particular limitation on the number of the imaging optical system 20 and the imaging element 22, and it may also be three or more.

[0036] The positional relationship of the optical elements (lenses, prisms, filters, and aperture stops) of the two imaging optical systems 20A, 20B with respect to the imaging elements 22A, 22B is constant. The imaging optical systems 20A, 20B are positioned such that the optical axes of the optical elements therein are orthogonal to the central portions of the light-receiving regions of the corresponding imaging elements 22, and the light-receiving regions become the imaging surfaces of the corresponding fisheye lenses.

[0037] In Figure 2 the illustrated embodiment, the imaging optical systems 20A, 20B have the same specifications, their optical axes are aligned with each other, and they are combined in the reverse direction. The imaging elements 22A, 22B convert the received light distribution into image signals and output image frames to the image processing block on the controller in sequence.

[0038] Specifically, as will be described in detail later, briefly speaking, the images captured by the imaging elements 22A, 22B respectively are subjected to synthesis processing to generate a 4π steradian spherical image (hereinafter referred to as an "omnidirectional image"). The omnidirectional image is a photographic image in all directions that can be seen from the shooting location. In the following embodiments, although the generation of the omnidirectional image is taken as an example for description, in addition to this, it may also be a panoramic image of only 360 degrees of the horizontal plane, or an image of a part of the omnidirectional or 360-degree horizontal panorama. The omnidirectional image can be either a still image or an animated image.

[0039] Figure 3(A) shows the hardware configuration of the omnidirectional imaging device 10 according to this embodiment. The omnidirectional imaging device 10 includes a CPU (Central Processing Unit) 112, a ROM (Read Only Memory) 114, an image processing module 116, a video compression module 118, a still image compression module 119, a DRAM (Dynamic Random Access Memory) 132 connected through a DRAM interface 120, and an acceleration sensor 136 connected through an external sensor interface 124.

[0040] The CPU 112 controls the operations of various parts and the overall operation of the omnidirectional imaging device 10. The ROM 114 stores a control program and various parameters written in a decryptable code by the CPU 112. The image processing module 116 is connected to two imaging elements 130A, 130B ( Figure 2 imaging elements 22A, 22B in).), and inputs the image signals of the images captured by each. The image processing module 116 includes an ISP (Image Signal Processor), etc., and performs brightness compensation, Bayer interpolation, white balance compensation, gamma correction, etc. on the image signals input from the imaging elements 130A, 130B. The image processing module 116 further performs a synthesis process on the multiple images obtained from the imaging elements 130A, 130B to generate the above-mentioned omnidirectional image.

[0041] The video compression module 118 is a codec module that performs video compression and decompression such as MPEG (Moving Picture Expert Group)-4 AVC (Advanced Video Coding) / H.264. The video compression module 118 is used to compress the video data of the generated omnidirectional image. The still image compression module 119 is a codec module that performs still image compression and decompression such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format). The still image compression module 119 is used to compress the still image data of the generated omnidirectional image. The DRAM 132 provides a storage area for temporarily storing data when performing various signal processing and image processing. The acceleration sensor 136 detects the three-axis acceleration components, and the detected acceleration components are used to detect the zenith compensation of the omnidirectional image in the vertical direction. Regarding the acceleration sensor 136, in addition to the acceleration sensor, it may also have the function of either or both of a three-axis angular rate sensor and an electronic compass.

[0042] The all-sky photography device 10 further includes an external storage interface 122, a USB (Universal Serial Bus) interface 126, a serial module 128, and a video output interface 129. The external storage interface 122 is connected to an external register 134. The external storage interface 122 is used to control the reading and writing in the external register 134 such as a memory card inserted into the memory card slot.

[0043] The USB interface 126 is connected to a USB connector 138. The USB interface 126 controls the USB communication with external devices such as an information terminal 50 and a personal computer connected through the USB connector 138. The serial module 128 is used to control the serial communication with external devices such as the information terminal 50 or a personal computer, and is connected to a wireless module 140. The video output interface 129 is an interface for connecting HDMI (High-Definition Multimedia Interface, HDMI is a registered trademark), etc., and can output the captured image to an external display for video.

[0044] After operating the electrical image source switch to turn on the electrical image source, the above control program is loaded into the main memory. The CPU 112 controls the operation of each part of the device according to the program read into the main memory, and temporarily stores the data required for control in the memory. Thus, the various functional parts and processes of the all-sky photography device 10 described later are realized.

[0045] Figure 3 (B) shows the hardware configuration of the information terminal 50 according to the present embodiment. Figure 3 The information terminal 50 shown in (B) of FIG. has a CPU 152, a RAM 154, a built-in register 156, an input device 158, an external register 160, a display 162, a wireless module 164, a USB connector 166, and a built-in camera 168.

[0046] The CPU 152 controls the operation of each functional part and the overall operation of the information terminal 50. The RAM 154 provides a working area for the CPU 152. The built-in register 156 stores various data such as application programs and images responsible for the operating system and the processing on the information terminal 50 side of the present embodiment written in a code that can be interpreted by the CPU 152.

[0047] The input device 158 is an input device such as a touch screen, and constitutes the input device of the present embodiment that provides a user interface. The external register 160 is a removable recording medium installed in a portable memory card slot, and is used to record various data such as image data and still image data in video format. The wireless module 164 establishes a connection for wireless LAN communication with external devices such as the all-sky photography device 10. The USB connector 166 establishes a USB connection with an external device.

[0048] In the following description, the wireless module 164 is used as the communication means for communicating with the omnidirectional imaging device 10. However, it is not limited to a specific specification, and external devices can also be connected through other wireless communications such as Bluetooth (registered trademark) or wireless USB, the USB connector 166, or wired communications such as a wired LAN (Local Area Network).

[0049] The display 162 performs the display of an operation screen for user operation, the display of a monitoring image of an image captured by the omnidirectional imaging device 10 before or during shooting, the playback of saved animations and still images, and the display for viewing. Through the display 162 and the input device 158, the user can issue a shooting instruction and make various setting changes to the omnidirectional imaging device 10 by means of the operation screen. The display 162 constitutes the built-in display device of the present embodiment. In addition, the information terminal 50 may also be provided with a video interface such as HDMI (registered trademark), and the screen is output to an external display device such as a monitor or a projector through the video interface.

[0050] The built-in camera 168 is a standard camera equipped in the information terminal 50, which captures ordinary still images or videos and generates still image data and video data. The built-in camera 168 constitutes the imaging device of the present embodiment.

[0051] When the power supply of the information terminal 50 is turned on and it is in the power-on state, various programs are read from the ROM and the built-in memory 156 and loaded into the RAM 154. The CPU 152 controls the operation of each part of the device according to the programs read into the RAM 154, and temporarily stores the data required for control in the memory, thereby implementing the various functional parts and processes of the information terminal 50 described below. The above application programs are programs of the present embodiment for operating the information processing device or computer device as each functional part. The above programs are installed in the information processing device or computer device, and the information terminal 50 of the present embodiment is implemented by executing these programs by the information processing device or computer device.

[0052] Regarding the hardware structure of the content disclosure server 70, detailed description is omitted here, but it has the same hardware configuration as the information terminal 50 shown in (B) of Figure 3 and the addition, deletion, or modification of hardware components can be made according to the usage.

[0053] The following further describes in detail the omnidirectional image content creation function of the present embodiment with reference to Figure 4 . Figure 4 It is a functional structure module diagram of omnidirectional image content creation implemented on the information terminal 50 according to the present embodiment.

[0054] As shown in Figure 4As shown in the figure, the functional module 200 of the information terminal 50 includes a communication unit 202, a built-in camera control unit 204, a display control unit 206, an input unit 208, a panoramic image content creation unit 220, and a content storage unit 240.

[0055] The communication unit 202 is used to control the wireless module 164 of the external device interface and perform communication with the panoramic photography device 10. The built-in camera control unit 204 is used to control the built-in camera 168 as the above-mentioned imaging device.

[0056] The display control unit 206 is used to control the display 162 as the display device. The input unit 208 is used to control the input device 158 as the input device.

[0057] In the best embodiment, the panoramic image content creation unit 220 is provided as an application program, which can integrate the function of controlling the panoramic photography device 10 to capture panoramic images and the photography function of the built-in camera 168 equipped with the information terminal 50, and uniformly manage various images (panoramic images and ordinary images) captured by these functions.

[0058] The panoramic image content creation unit 220 includes a panoramic photography device control unit 222, a built-in photography function enhancement unit 226, an image information management unit 230, and a UI (user interface) unit 234.

[0059] The panoramic photography device control unit 222 controls the external panoramic photography device 10 through the communication unit 202. The panoramic photography device control unit 222 issues instructions to the panoramic photography device 10 to make the panoramic photography device 10 perform panoramic image capture and send the captured panoramic image to the information terminal 50. The panoramic photography device control unit 222 obtains the panoramic image from the panoramic photography device 10 through the communication unit 202. The panoramic photography device control unit 222 constitutes the external device control unit and the first acquisition unit in this embodiment. The panoramic image can be either a still image or an animated image, but in the following description, the still image is taken as the center for explanation.

[0060] The built-in photography function invocation unit 226 invokes the shooting function of the built-in camera 168 through the built-in camera control unit 204. The built-in photography function invocation unit 226 also obtains the photographic image captured by the built-in camera 168 from the built-in camera control unit 204. The built-in photography function invocation unit 226 constitutes the invocation unit and the second acquisition unit in this embodiment.

[0061] Here are three installation methods that can be used to directly or indirectly control the built-in camera 168 from an application. The first method is to directly call the camera application that uses the built-in camera 168 for shooting. The second method is to call the camera function that uses the built-in camera 168 for shooting. The third method is to directly install the function for controlling the built-in camera 168 within the omnidirectional image content creation unit 220. Although, from the perspective of user operability, the second and third methods are better, the above three methods are the same in the sense of directly or indirectly controlling the built-in camera 168 from an application.

[0062] Figure 4 Supporting the above three methods, the built-in photography function calling unit 226 calls the built-in camera control unit 204, which is a camera function, to call the shooting function of the built-in camera 168. However, the present invention is not limited thereto.

[0063] As the image obtained by shooting with the built-in camera 168, in the following embodiments, ordinary still images are taken as an example, but it can also be a moving image. Moreover, in the following embodiments, when the omnidirectional image is saved in an image folder (album, camera roll), the built-in camera 168 for shooting ordinary images can be used.

[0064] The image information management unit 230 manages by associating the omnidirectional image obtained by the omnidirectional photography device control unit 222 with the ordinary image obtained by the built-in photography function calling unit 226, and saves the omnidirectional image content into the content storage unit 240. The omnidirectional image content includes one or more omnidirectional images and one or more photographed images. The image information management unit 230 constitutes the management device of this embodiment. The image information management unit 230 can also, at the same time as saving to the content storage unit 240, or instead of saving, send the created omnidirectional image content (i.e., upload) to the external content disclosure server 70 and log in.

[0065] Figure 4 The shown image information management unit 230 includes an associated position specifying unit 232. The associated position specifying unit 232 can specify a specified position in the omnidirectional image shot by the omnidirectional imaging device 10 as an associated object associated with the above ordinary image, etc. The associated position specifying unit 232 is the position specifying unit of this embodiment.

[0066] In a specific embodiment, the associated position specifying unit 232 can specify the central area of the display area when the omnidirectional image is displayed on the display 162 as the specified position in the omnidirectional image. In another embodiment, the associated position specifying unit 232 can specify the specified area specified by the above input device 158 in the display area when the omnidirectional image is displayed on the display 162 as the specified position in the omnidirectional image.

[0067] In addition to associating the above-mentioned ordinary image with the designated position in the omnidirectional image, the Image Information Management Unit 230 can also associate it with the content of the upper unit including the omnidirectional image. Although not shown in Figure 4 the display, in addition to the ordinary image obtained by the Built-in Photography Function Call Unit 226, the Image Information Management Unit 230 can also manage, in association with the omnidirectional image or the content, one or both of the images stored in the built-in register 156 or the external register 160 of the information terminal 50 and the externally acquired images received from an image management system (such as a content disclosure server or other services) that can communicate with the information terminal 50. Furthermore, the Image Information Management Unit 230 can manage by associating the designated position of the second omnidirectional image with the first omnidirectional image. Similarly, it can also manage by associating the designated position of the first omnidirectional image with the second omnidirectional image.

[0068] The UI Unit 234 manages the transition of the screen for creating the content of the omnidirectional image and provides the screen and GUI (Graphical User Interface) components such as buttons that make up the screen. As a GUI component, the UI Unit 234 constitutes the first providing unit and the second providing unit for providing the first indication unit and the second indication unit in the present embodiment. The first indication unit is used to receive an instruction to have the omnidirectional photographing device 10 take a photograph, and the second indication unit is used to receive an instruction to call the photographing function of the built-in camera 168.

[0069] The following refers to Figures 5 to 18 for a detailed description of the omnidirectional image content creation process of the present embodiment. Figure 5 and Figure 6 and Figure 7 and Figure 9 and Figure 11 are the flowcharts of the omnidirectional image content creation process executed by the information terminal 50 of the present embodiment. It should be noted that Figure 5 and Figure 6 and Figure 7 and Figure 9 and Figure 11 show the main processes related to associating multiple images in the present embodiment, while omitting the processes of other functions and exception handling. Figures 14 to 18 is a schematic diagram of various screens displayed on the display 162 of the information terminal 50 of the present embodiment.

[0070] Figure 14 In (A), the main screen 400 of the omnidirectional image content creation application is shown. The "Content Creation" button 402 for starting the creation of omnidirectional image content is set on the main screen 400 shown in (A). Figure 5 and Figure 6 The processes shown are triggered, for example, when the user clicksFigure 14 On the main screen 400 shown in (A), the "Content Creation" button 402 is clicked. Figure 5 The process starts at step S100 shown in FIG.

[0071] In step S101, the information terminal 50 displays Figure 14 The content information input screen 410 shown in (B) of FIG. Figure 14 In the content information input screen 410 shown in (B), information about the content, such as the content name or any English letters representing the content, can be input using text boxes 412, 414, etc. An "Insert Picture" button 416 for inserting a room map is also provided on the content information input screen 410. The user can click the "Insert Picture" button 416 to call up the process of inserting a specified room map image into the content. Furthermore, the content information input screen 410 is provided with an "OK" button 418 for ending the input of basic information of the content and proceeding to the next operation of creating the content.

[0072] In step S102, at the information terminal 50 Figure 14 In step S102, when it is determined that the "Insert Picture" button 416 is clicked and the room type map insertion instruction is accepted (insert room type map), the process proceeds to step S103.

[0073] In step S103, the information terminal 50 Figure 14 As shown in (C), an image source selection menu for accepting selection of the room plan image source is displayed. The image source selection menu includes a "shoot with all-sky camera" button 422 for selecting the built-in camera 168 as the image source, and a "select all-sky image from album" button 424 for selecting an image folder in the information terminal 50. When one of the buttons is clicked, the process proceeds to step S104.

[0074] Although this embodiment Figure 14 The screen 410 shown in (C) only displays the "shoot with spherical camera" button 422 and the "select spherical image from album" button 424, but a button for selecting an external image management system on the network can also be set as the image source.

[0075] In step S104, the information terminal 50 performs room map registration processing. Here, the room map is a basic image of a series of associated objects of the spherical image in the application. In addition, as a basic image that is mutually associated with the content as a whole, the present embodiment uses the room map (also called floor map and schematic diagram, etc.) as the basic image. However, it is also possible to use an image other than the room map as the basic image, or there is no room map, and the entire content is composed of two images of the spherical image and the ordinary image.

[0076] Figure 7 This is a flowchart of the room type diagram login process. Figure 7 The processing shown Figure 5 is called in the processing of step S104 shown and starts processing from step S201. In step S202, the information terminal 50 determines whether to select photographing with the built-in camera 168. In step S202, if it is determined that the "Take a photo with the omnidirectional camera" button 422 in the image source selection menu is clicked and photographing with the built-in camera 168 is selected (Yes), the processing proceeds to step S203.

[0077] In step S203, the information terminal 50 calls the built-in camera control unit 204 through the built-in photographing function calling unit 226. Figure 14 (D) shows the photographing screen 430 for photographing the room type diagram with the built-in camera, which is displayed with the call of the built-in camera control unit 204. In Figure 14 the photographing screen 430 shown in (D), when the photographing button 432 is clicked and after the photographing process is completed, the processing proceeds to step S204. In step S204, the information terminal 50 acquires a normal image taken with the built-in camera 168.

[0078] On the other hand, in step S202, if it is determined that photographing with the built-in camera is not selected (No), the processing proceeds to step S207. In step S207, the information terminal 50 determines whether the information terminal 50 is selected as the image source. In step S207, when it is determined that the "Select omnidirectional image from album" button 424 is selected from the image source selection menu and the information terminal 50 is selected (Yes), the processing proceeds to step S208.

[0079] In step S208, the information terminal 50 reads the list of images saved in the information terminal 50 (such as the image folder) and selectively displays the list. In step S209, the information terminal 50 accepts the selection from the list of images in the information terminal 50, acquires the selected image, and the processing proceeds to step S205.

[0080] In step S205, the information terminal 50 determines whether an image has been acquired (acquired by photographing, or acquired from internal or external). In step S205, if it is determined that an image has been acquired (Yes), the processing proceeds to step S206. In step S206, the information terminal 50 logs in the acquired image as the room type diagram, associates it with the content, records it as "having" the room type diagram, and returns in step S210 Figure 5 to the processing. In this case, the control returns to step S101, and the updated content information input screen is displayed.

[0081] Figure 15(A) shows the updated content information input screen 410 when the room type diagram is set. In the content information input screen 410 shown in this figure, a new room type diagram image 415 is displayed. The room type diagram in this embodiment is an image taken by the built-in camera 168. However, this is a picture taken on hand, not limited to photos, and can also be an image such as a pre-drawn picture or illustration. This image can also be an image saved in an image folder (album).

[0082] On the other hand, if in step S205, it is determined that no photography or image selection is performed, etc., and thus no image is obtained (No), the process directly goes to step S210 and returns Figure 5 to the process. In addition, if in step S207 it is determined that an area other than buttons 422 and 424 is clicked in the incoming image source selection menu, etc., and all image sources are not selected (No), the process directly goes to step S210 and returns Figure 5 to the process. Here, the image source selection menu can also be configured to obtain images obtained from the outside from an image management system (such as the content disclosure server 70 or other services) capable of communicating with the information terminal 50 in addition to buttons 422 and 424.

[0083] Return Figure 5 , in step S102, when it is determined that in Figure 14 the (B) of Figure 15 or when the "OK" button 418 is clicked on the content information input screen 410 shown in the (A) of

[0084] and an instruction to proceed to the next process for creating content is accepted (next screen), the process goes to step S105.

[0085] In step S105, the information terminal 50 performs different processes according to whether there is a room type diagram. If it is determined in step S105 that there is a "room type diagram" (Yes), the process goes to step S106. Figure 15 In step S106, the information terminal 50 displays a content creation screen (with room type diagram) 440 in order to register the entire sky image in association with the position on the room type diagram. In step S107, the information terminal 50 performs different processes according to the instructions received from the user on the content creation screen (with room type diagram) 440. In the state where the room type diagram is registered and no entire sky camera image is registered at all, the content creation screen (with room type diagram) 440 becomes as

[0086] Figure 15In the content creation screen (with floor plan) 440 shown in (B), a display area 442 for the floor plan image is provided. When a position specifying operation such as a user click is detected in this display area 442, the position coordinates related to the operation on the display area 442 can be obtained. Additionally, as an arbitrary configuration, it can be configured to magnify or reduce the floor plan image displayed in the display area 442 with operations such as pinch-in and pinch-out.

[0087] Figure 8 is a schematic diagram for explaining a method related to positions on the floor plan. The display area 442 is displayed on the display with a specified size, and an image of the floor plan is set within this display area 442 at a specified ratio. Therefore, the specified position 302 in the display area 442 can be Figure 8 converted from the relative positions (X1 / (X1 + X2), Y1 / (Y1 + Y2)) shown to the coordinates on the floor plan image. After the display area 442 on the content creation screen (with floor plan) 440 is clicked or the like, the coordinate values of the position on the floor plan image corresponding to the specified position operated on in the display area 442 are determined, and an acceptance of the login instruction for the omnidirectional image at this position is obtained. Figure 8 Taking the upper left corner as the origin, the coordinate value range is 0 to 1. However, the method of setting the origin is not limited, and the range of the coordinate values is also not limited.

[0088] In the content creation screen (with floor plan) 440 shown in step S106, in addition to the login of the omnidirectional image ("Login"), "Save" of the content and confirmation of the omnidirectional image that has been logged in the content ("Confirm") can also be called. However, in a state where there is no omnidirectional image logged in at all, the GUI component for "Confirm" will not be displayed. On the other hand, the GUI component for "Save" the content can also be displayed for temporary saving.

[0089] When it is determined in step S107 that "Login" has been accepted on the content creation screen (with floor plan) 440, the process proceeds to step S108. In step S108, the information terminal 50 displays Figure 15 the image source selection menu shown in (C). The image source selection menu is provided with a "Shoot with Omnidirectional Camera" button 452 for selecting the external omnidirectional photography device 10 as the image source and a "Select Omnidirectional Image from Album" button 454 for selecting the image folder within the information terminal 50. After one of the buttons is clicked, the process proceeds to step S109. The "Shoot with Omnidirectional Camera" button 452 constitutes the first indication part of this embodiment.

[0090] In step S109, the information terminal 50 sets the content and the position coordinate values on the floor plan image corresponding to the specified position as the associated objects of the omnidirectional image to be logged in. In step S110, the information terminal 50 executes the omnidirectional image login process.

[0091] On the other hand, in step S105, if it is determined that there is no floor plan (No), the process proceeds to step S111. In step S111, the information terminal 50 displays the content creation screen (without floor plan) 460. In step S112, the information terminal 50 performs different processes according to the user instructions received in the content creation screen (without floor plan) 460. In addition, in a state where no floor plan is logged in and no omnidirectional camera images are logged in at all, the content creation screen (without floor plan) 460 becomes the state shown in Figure 15 (D) as shown. Figure 15 The content creation screen (without floor plan) 460 shown in (D) as shown is the same as the screen shown in Figure 15 (C) and is provided with an "omnidirectional camera shooting" button 462 and a "select omnidirectional image from album" button 464. After clicking any one of these buttons, an instruction for logging in the omnidirectional image ("login") is received, and the process proceeds to step S112.

[0092] In addition, in the content creation screen (without floor plan) 460 shown in step S111, in addition to logging in the omnidirectional image ("login"), "saving" of the content and confirmation ("confirmation") of the omnidirectional images already logged in the content can also be called. However, in a state where no omnidirectional images are logged in, the GUI component for "confirmation" is not displayed, and the GUI component for temporarily saving the content can be displayed for temporary saving.

[0093] If it is determined in step S112 that the content creation screen (without floor plan) 460 has accepted "login", the process proceeds to step S113. In step S113, the information terminal 50 sets the content itself as the associated object of the omnidirectional image to be taken, and in step S110, the information terminal 50 performs the omnidirectional image login process.

[0094] In addition, in the Figure 15 (C) and Figure 15 (D) shown in the figure, in this embodiment, only the buttons 452, 454 and the buttons 462, 464 are respectively displayed, but as the image source, buttons for selecting omnidirectional images that have been taken and not yet transmitted within the omnidirectional photography device 10 and buttons for selecting omnidirectional images in an external image management system on the network can be selected.

[0095] Figure 9 is the flowchart of the omnidirectional image login process. Figure 9 The process shown is called in the process of step S110 shown in Figure 5 and starts from step S300. In step S301, the information terminal 50 determines whether shooting with an external omnidirectional photography device 10 is specified. In step S301, when it is determinedFigure 15 The button 452 of the image source selection menu on the screen 440 shown in (C) is clicked, or Figure 15 The button 462 on the screen 460 shown in (D) is clicked, or when it is selected to shoot with the omnidirectional camera device 10 (yes), the process proceeds to step S302.

[0096] In step S302, the information terminal 50 controls the omnidirectional camera device 10 through the omnidirectional camera device control unit 222 to shoot an omnidirectional image. Figure 16 (A) is the omnidirectional camera screen 470 for shooting by the omnidirectional camera device 10. On the omnidirectional camera screen 470, when the shooting button 472 is clicked, a shooting instruction is sent from the omnidirectional camera device control unit 222 to the omnidirectional camera device 10 via the communication unit 202, and the shooting process is executed in the omnidirectional camera device 10. After the shooting is completed, the image starts to be transmitted to the information terminal 50. Figure 16 (B) is the transmission screen 480 where the omnidirectional image is being transmitted from the omnidirectional camera device 10 to the information terminal 50. In step S303, the information terminal 50 obtains the omnidirectional image transmitted from the omnidirectional camera device 10 of the external camera device through the omnidirectional camera device control unit 222, and then the process proceeds to step S313.

[0097] On the other hand, in step S301, if it is determined that shooting with the omnidirectional camera device 10 is not selected (no), the process proceeds to step S304. In step S304, the information terminal 50 determines whether an image that has been shot in the omnidirectional camera device 10 of the external camera device is selected as the image source. If it is determined in step S304 that the image that has been shot in the omnidirectional camera device 10 is selected as the image source (yes), the process proceeds to step S305.

[0098] In step S305, the information terminal 50 obtains a list of omnidirectional images that have been shot (and not yet transmitted to the information terminal 50) from the omnidirectional camera device 10 through the omnidirectional camera device control unit 222, and displays the list for selection. In step S306, the information terminal 50 accepts the selection of the specified image from the list of omnidirectional images in the omnidirectional camera device 10, receives the selected omnidirectional image from the omnidirectional camera device 10 in step S303, and the process proceeds to step S313.

[0099] On the other hand, in step S304, if it is determined that an image that has been shot in the external omnidirectional camera device 10 is not selected (no), the process proceeds to step S307. In addition, although in Figure 15 the menu shown in (C) and Figure 15When there is a button for selecting a panoramic image that has been captured (and not yet transmitted to the information terminal 50) within the panoramic photography device 10 as the image source in the screen 460 shown in (D), the determination in step S304 is performed, but this can also be omitted and the process can directly proceed to step S307.

[0100] In step S307, the information terminal 50 determines whether an image within the information terminal 50 has been selected as the image source. If it is determined in step S307 that the information terminal 50 has been selected as the image source (Yes), the process proceeds to step S308.

[0101] In step S308, the information terminal 50 reads the list of panoramic images within the information terminal 50. The panoramic images are in the same format as general images such as JPEG, but are accompanied by meta-information for identifying whether they are panoramic images. Based on this meta-information, if an image that is not a panoramic image is selected, other images can be prompted for selection. Or the panoramic images and other generally captured images in the obtained image list can be distinguished and displayed, or only panoramic images can be extracted in advance and included in the displayed list. Additionally, the panoramic images that have been captured and delivered are set to be saved within the information terminal 50. In step S309, the information terminal 50 accepts the selection from the list of panoramic images within the information terminal 50, reads the selected image, and the process proceeds to step S313.

[0102] On the other hand, if it is determined in step S307 that the information terminal 50 has not been selected (No), the process proceeds to step S310. In step S310, the information terminal 50 obtains a list of panoramic images from an image management system (such as the content disclosure server 70) that can communicate with the information terminal 50 via the network. The panoramic images that have been captured are uploaded in this set image management system. In step S311, the information terminal 50 accepts the selection from the list of panoramic images within the image management system, and in step S312, receives the panoramic image via the network, and the process proceeds to step S313.

[0103] In Figure 15 the image source selection menu shown in (C) and Figure 15 when there is a button for selecting an image management system on the network as the image source on the screen 460 shown in (D), the processes in steps S310 to S312 are performed, but these processes can also be omitted, and in this case, the process directly proceeds to step S316.

[0104] In step S313, the information terminal 50 displays a screen for determining the panoramic image to be used. In step S314, the information terminal 50 determines whether the panoramic image to be used has been determined. Figure 16The (C) of this is the omnidirectional image determination screen 490 for determining the omnidirectional image. On the omnidirectional image determination screen 490, there are provided a button 494 for determining to use the currently displayed omnidirectional image and a retry button 492 for re - shooting or re - selecting.

[0105] In step S314, if it is determined that the button 494 has been clicked on the omnidirectional image determination screen 490 and the image has been determined (Yes), the process proceeds to step S315. In step S315, the information terminal 50 associates the determined omnidirectional image with the associated object set in step S109 or S13 in Figure 5 and then logs in the content, and in step S316, returns to Figure 5 the process. In addition, the login of this associated object can be moved to any position afterwards. On the other hand, if it is determined in step S314 that Figure 16 the button 492 has been clicked or retry has been specified (No) on the omnidirectional image confirmation screen 490 shown in the (C) of this, the process returns to step S301. Here, the retry is described in the state of selecting the same image source, but the retry can start from the selection of the image source.

[0106] Figure 9 After the omnidirectional image login process shown is completed, it returns to step S105 again, and in step S106 or step S111, the updated content creation screen is displayed.

[0107] Here, in the case of the state where the room type map is not logged in and no omnidirectional photography image is logged in after being updated in step S111, the content creation screen is as Figure 16 shown in the (D) of this.

[0108] Figure 16 In the content creation screen (without room type map) 460 shown in the (D) of this, there are provided a display area 466 for displaying the omnidirectional image and a "Save" button 468 for saving the content. The display area 466 is provided with a GUI component 465 for calling the screen for inputting the part for shooting the omnidirectional image. The content creation screen (without room type map) 460 is also provided with a "Select Omnidirectional Image from Album" button 464 and an icon 469 for shooting with the omnidirectional photography device.

[0109] The above - mentioned display area 466 for the omnidirectional image is a GUI component for calling to confirm the logged - in omnidirectional image ("Confirm"). The "Save" button 468 is a GUI component for calling the content saving process. By logging in one or more omnidirectional images, "Confirm" is displayed, and the "Save" GUI component of the content is displayed as needed.

[0110] Figure 18The (D) shows the content creation screen when updated through step S106, with a floor plan logged in and in a state where two omnidirectional photography images have been logged in.

[0111] Figure 18 The content creation screen (with floor plan) 440 shown in (D) shows the floor plan display area 442 located above the omnidirectional photography image display area 446, which is the same as Figure 16 the content creation screen (without floor plan) 460 shown in (D). Figure 18 The (D) shows the state where omnidirectional images logged in are associated with two positions on the floor plan. Markers 442a and 442b are marked at two positions in the floor plan display area 442. Moreover, Figure 18 the (D) shows the omnidirectional image corresponding to the position marker 442b marked with "①" among the two markers 442a and 442b being selected.

[0112] The display area 446 shows the selected omnidirectional image and is provided with a GUI component 445 for calling the input shooting part screen. A sector icon 442c indicating the angle during the display of the omnidirectional image is also overlapped on the floor plan display area 442. The omnidirectional image display area 446 is a GUI component for calling the confirmation ("confirmation") of the logged-in and selected omnidirectional image. Similar to Figure 16 the (D), Figure 18 the content creation screen (with floor plan) 440 shown in (D) is provided with a "Save" button 448.

[0113] The following uses Figure 16 the content creation screen (without floor plan) 460 shown in (D) to illustrate the implementation method of assigning ordinary image annotations to omnidirectional images. As described above, the content creation screen (with floor plan) 440 also implements the same content.

[0114] In Figure 16 the content creation screen (without floor plan) 460 shown in (D), when the display area 466 corresponding to the selected omnidirectional image is clicked, an instruction for image "confirmation" is accepted. When step S107 or step S112 determines that "confirmation" has been accepted in the content creation screen 440 / 460, the process proceeds to Figure 5 point B shown in Figure 6 i.e., proceeds to step S114 shown in

[0115] In step S114, the information terminal 50 displays Figure 17 the image confirmation screen 500 for confirming the specified omnidirectional image shown in (A). Figure 17The image confirmation screen 500 shown in (A) of [] contains a "Return" button 502 (represented by "←"), a display area 504 for displaying a specified all-sky image, and a menu button 508. On this screen, the displayed part in the all-sky image displayed in the display area 504 can be moved along with an operation indicating a direction such as a flick.

[0116] In Figure 17 in the image confirmation screen 500 shown in (A) of [], if the menu button 508 is clicked, then as shown in Figure 17 (B) of [], a menu containing an "Add Annotation" button 510 for performing image annotation is displayed in the image confirmation screen 500. After the "Add Annotation" button 510 is clicked, an instruction to add an annotation is accepted. In Figure 17 in the image confirmation screen 500 shown in (A) of [], if the "Return" button 502 is clicked, then an instruction to return to the content creation screen (for example, Figure 16 the content creation screen 460 in (D) of []) is accepted.

[0117] In step S115, if it is determined that an "Add Annotation" instruction has been accepted in the image confirmation screen 500 shown in Figure 17 (B) of [], the process proceeds to step S116. In step S116, the information terminal 50 determines the associated position through the associated position specifying unit 232. When adding an annotation is called through the menu button 508 in the image confirmation screen 500 shown in Figure 17 (B) of [], the position corresponding to the approximate center of the currently displayed part in the display area 504 on the all-sky image is determined as the associated position for adding an annotation.

[0118] Although in the above description, Figure 17 the "Add Annotation" button 510 is displayed by clicking the menu button 508 in the image confirmation screen 500 shown in (A) of [], it is not limited to this, and it is also possible to display the "Add Annotation" button 510 in the image confirmation screen 500 shown in Figure 17 (A) of [] as the specified position (such as 505) in the display area 504 is long-pressed. When adding an annotation is called through the area in the display area 504, the position on the all-sky image corresponding to the position when the area in the display area 504 is clicked is determined as the associated position for adding an annotation.

[0119] Figure 10 is a schematic diagram for explaining a method associated with a position on the all-sky image. As shown in the data structure 330 in Figure 10 (), the all-sky image is generated in a format represented by longitude from -180 degrees to +180 degrees on the horizontal axis and latitude from -90 degrees to +90 degrees on the vertical axis.

[0120] When the full celestial sphere image is displayed on a flat device such as the display 162 of the information terminal 50, the full celestial sphere image is pasted on the spherical object 310, and a virtual camera 312 is set at, for example, the center of the spherical object 310, and the projection image 314 observed from the virtual camera 312 is displayed. That is, the display area 504 is displayed on the display with a specified size, and the projection image 314 generated based on the full celestial sphere image is placed in the display area 504. The position on the full celestial sphere image corresponding to the center position of the projection image 314 being displayed can be obtained by the direction of the virtual camera 312. On the other hand, any position (for example, the clicked position) 322 in the display area 504 can be obtained from Figure 10 The relative position (X1 / (X1+X2), Y1 / (Y1+Y2)) shown is converted into coordinates 332 on the full celestial sphere image format 330 through a point 316 in the range corresponding to the projection image 314 on the spherical object 310. Figure 10 With the center as the origin, the coordinate values ​​are within the range of longitude -180 degrees to +180 degrees and latitude -90 degrees to +90 degrees, but there is no restriction on the method of setting the origin, and the range is also not limited. For example, it can also be expressed as longitude 0 degrees to 360 degrees and latitude 0 degrees to 180 degrees. In addition, when the wide-angle image is an image other than the full celestial sphere image and has a viewing angle of more than 180 degrees, the coordinates of the image can be set in a form that conforms to the viewing angle to replace the coordinates of the above-mentioned full celestial sphere image.

[0121] Refer again Figure 6 In step S117, the information terminal 50 sets the identified position in the full celestial image as the relevant target. In step S118, the information terminal 50 displays Figure 17 The comment setting screen 520 shown in (C) . Figure 17 The annotation setting screen 520 shown in (C) is provided with a "select image" button 522. By clicking the "select image" button 522, the user can call up a process of taking an image by the built-in camera 168 of the information terminal 50. In addition, as for the annotation, only adding an image is exemplified here, but a text input field can also be provided in conjunction with the image to add text.

[0122] In step S119, the information terminal 50 will display an image source selection menu for receiving a conventional image source selection, such as Figure 17 The image source selection menu includes a "shoot with camera" button 524 for selecting the built-in camera as the image source, and a "select image from album" button 526 for selecting the image folder in the information terminal 50 as the image source. When any button is clicked, the process proceeds to step S110. In addition, the "shoot with camera" button 524 constitutes the second indication unit in this embodiment.

[0123] In step S120, the information terminal 50 performs ordinary image registration processing.

[0124] Figure 11 It is a processing flow chart for registering ordinary images. Figure 11 The processing shown starts from step S400 after being called in the processing of step S120 shown in Figure 5 In step S401, the information terminal 50 determines whether shooting with the built-in camera is specified. In step S401, if it is determined that the button 524 is clicked in the image source selection menu of (D) in Figure 17 and shooting with the built-in camera 168 is selected (yes), the processing proceeds to step S402.

[0125] In step S402, the information terminal 50 calls the built-in camera control unit 204 through the built-in photography function call unit 226. Figure 18 (A) in is the screen 530 for shooting ordinary images with the built-in camera displayed along with the call of the built-in camera control unit 204. In Figure 18 In the shooting screen 530 shown in (A) in, when the shooting button 532 is clicked and the shooting process ends, the processing proceeds to step S410. In step S403, the information terminal 50 acquires the ordinary image shot by the built-in camera 168, and the processing proceeds to step S410.

[0126] On the other hand, in step S401, if it is determined that shooting with the built-in camera 168 is not selected (no), the processing branches to step S404. In step S404, the information terminal 50 determines whether the information terminal 50 is selected as the image source. In step S404, if it is determined that the information terminal 50 is selected as the image source (yes), the processing proceeds to step S405.

[0127] In step S405, the information terminal 50 reads the image list in the information terminal 50 and displays it in a list. In step S406, the information terminal 50 accepts the selection in the image list in the information terminal 50, reads the selected image, and the processing proceeds to step S410.

[0128] On the other hand, in step S404, if it is determined that the inside of the information terminal 50 is not selected (no), the processing proceeds to step S407. In step S407, the information terminal 50 acquires the ordinary image list from the ordinary image list through the network via an image management system (such as the content disclosure server 70) that can communicate with the information terminal 50. In step S408, the information terminal 50 accepts the selection in the ordinary image list in the image management system, and in step S409, receives the ordinary image through the network, and the processing proceeds to step S410. The processing from steps S407 to 409 is in Figure 17The process implemented when there is a button for selecting the image management system as the image source in the menu shown in (D) can be omitted. In this case, the process can directly proceed to step S413.

[0129] In step S410, the information terminal 50 displays a screen for determining the ordinary image to be used. In step S410, the information terminal 50 determines whether the ordinary image to be used has been determined. Figure 18 The image determination screen 540 for determining the ordinary image is shown in (B). The image determination screen 540 has a button 542 for determining to use the currently displayed image and a button 544 for reshooting or selecting.

[0130] In step S411, if the button 542 on the image determination screen 540 is clicked and it is determined that the image has been determined (Yes), the process proceeds to step S412. In step S412, the information terminal 50 manages and logs in the determined ordinary image with the associated object image, and in step S413, it returns to Figure 6 the process. The associated object here is the coordinate value of the specified position on the all-sky image determined in step S117 or the direction indicating the specified position. The login of this associated object can be moved to any position arbitrarily afterwards. On the other hand, if the button 544 on the image determination screen 540 is clicked in step S411 and it is determined that reshooting is specified (No), the process returns to step S401.

[0131] Return Figure 6 After the process returns, it returns to the process of step S114 through point B. In step S114, the information terminal 50 displays an updated image confirmation screen 500 as shown in Figure 18 (C). In Figure 18 the image confirmation screen 500 shown in (C), a marker 550 indicating the position associated with the ordinary image is displayed in the display area 504 of the all-sky image. The coordinates of this marker 550 correspond to the coordinates set when the ordinary image is associated and logged in as the associated object in step S412. In addition, since there is more than one annotation, a "Delete" button 552 and an "Edit Annotation" button 554 for deleting the annotation are displayed, but there is no further processing regarding the deletion and editing of the annotation.

[0132] In step S115, if it is determined that the information terminal 50 Figure 17 in (A) or Figure 18 the "←" button 502 indicating return in the image confirmation screen 500 shown in (C) is clicked and it returns to the upper level, it proceeds to Figure 5 step S105 shown. In this case, it is necessary to further execute the process of associating and logging in other all-sky images related to the specified position on the floor plan of the content or the content itself.

[0133] Here, it becomes Figure 16 the state where the content creation screen (without floor plan) 460 shown in (D) or the content creation screen (with floor plan) 440 where the omnidirectional image is logged in the designated position of the floor plan. In step S107 or step S112, when it is determined that the "save" instruction has been received in the content creation screen 440 / 460, it proceeds to Figure 6 the step S121 shown in.

[0134] In step S121, the information terminal 50 saves the created omnidirectional image content to the content storage unit 240, proceeds to step S122, and ends this process. Instead of saving to the content storage unit 240, it is also possible to upload the omnidirectional image content to the content disclosure server 70.

[0135] The specific processing is not involved here. In short, it is also possible to have functions such as deleting and replacing the already logged omnidirectional images and ordinary images, blurring, stamping, scaling, etc. of the designated areas of the omnidirectional images and ordinary images, inputting, changing, and deleting the names of the content and omnidirectional images before saving and uploading the omnidirectional image content.

[0136] In the above description, the ordinary image is associated with the designated position in the omnidirectional image as a note, but it is not limited to the ordinary image. Other omnidirectional images can also be added, such as adding other types of information such as text and sound as notes. In addition, although the image is associated with the designated position in the omnidirectional image, it can be associated with the omnidirectional image rather than the designated position, or it can be associated with the content of the upper unit including the omnidirectional image.

[0137] For example, in the content introducing the real estate house and equipment, the ordinary images of each room's equipment are associated with the designated positions in the omnidirectional images of each room. On the other hand, there are also ordinary images such as those taken from the entrance and balcony views. In this case, the ordinary images can be associated with the entire content rather than the omnidirectional images of each room. In addition, the omnidirectional images of each room are associated with each other at the designated position (for example, the position corresponding to the door).

[0138] Figure 12 It is a schematic diagram of the data structure for managing the omnidirectional image content generated by the information terminal 50 of this embodiment. Figure 12 In, (A) represents the content information management table that manages the correspondence between the content identifier (ID) and information such as the name, (B) represents the omnidirectional image management table that manages the correspondence between the omnidirectional image ID and its storage location, and (C) represents the ordinary image management table that manages the correspondence between the ordinary image ID and its storage location.

[0139] Figure 12 (D) shows a content element management table that manages by correlating the ID, name, content ID, associated omnidirectional image ID, and coordinates on the room layout diagram of content elements such as room IDs with omnidirectional images. Here, as described previously, the coordinates on the room layout diagram are obtained and saved as shown in Figure 8 . In the management table, a null value indicates the absence of a value. That is, content ID = 102 indicates a room (content element) that does not contain a room layout diagram or whose position on the room layout diagram is not bound.

[0140] Figure 12 (E) represents a room layout diagram management table that manages by correlating the room layout diagram ID, content ID, and ordinary image ID that identifies the ordinary image corresponding to the room layout diagram. Figure 12 (F) shows a note management table that manages by correlating the note ID, the omnidirectional image ID associated with the note, the coordinates on the omnidirectional image where the note is placed, the note description, and the ordinary image ID that identifies the ordinary image added as a note. Here, the coordinates on the omnidirectional image can be obtained as described in Figure 10 . The note management table can also be configured to associate the entire content instead of associating with the omnidirectional image.

[0141] Figure 13 is a schematic diagram of the file structure when the omnidirectional image content generated by the information terminal 50 of the present embodiment is output. In Figure 13 , files of room layout diagram images, files of overlooking images, and each folder with an omnidirectional image name are set under a folder with a content name. Files of omnidirectional images and files of ordinary images associated with the omnidirectional images are placed in all folders with an omnidirectional image name.

[0142] Through the above processing, an omnidirectional image content with ordinary images associated with any position in the omnidirectional image and having a data structure as shown in Figure 12 can be created. Moreover, when viewing the omnidirectional image content on the viewfinder, the ordinary image can be embedded as a note in the omnidirectional image for display. For example, in the viewfinder, when a specified operation (such as clicking, single-clicking, etc.) is performed on an image element (mark, icon, symbol) set at the note position in the omnidirectional image, the ordinary image associated with that position will pop up for display.

[0143] As described above, according to the above-described embodiments, in an information processing apparatus (computer apparatus) having a photographing unit, the function of controlling the photographing of an external photographing apparatus and the function of photographing with the photographing unit provided in the above information processing apparatus (computer apparatus) can be integrally controlled, various captured images can be uniformly managed, a method for implementing the information processing apparatus, and a storage medium and a computer apparatus storing a program for implementing the information processing apparatus.

[0144] In the above-described embodiment, the function of controlling the photographing of an external photographing apparatus and the function of photographing with the photographing unit provided in the information processing apparatus itself are integrated into the same application program. For this reason, when a user uses the function of controlling the photographing of an external photographing apparatus and the function of photographing with the photographing unit provided in the information processing apparatus itself, the user does not need to use different application programs.

[0145] When each photographing function is implemented by different application programs, it is necessary to temporarily save an image to an image folder such as an album, and then log in to the content through the image folder. In this case, the image folder stores images saved in chronological order and also stores images that should have been logged in to different contents in a mixed manner, so it is difficult to correctly log in the images to the content. In contrast, in the present embodiment described above, various images can be uniformly managed, so that the confusion that occurs when passing through the image folder can be avoided.

[0146] By integrating the all-sky image photographing for content creation and the ordinary image photographing, the workload of content creators can be reduced. Since content creators can perform ordinary image photographing on the content creation application installed in the information terminal 50, as long as the all-sky photographing apparatus 10 and the information terminal 50 are available, there is no longer a need for ordinary photographic equipment such as a camera, and there is no longer a need for the conventional ordinary image or video shooting operation. Furthermore, since the association between the content, the floor plan (appropriate), the all-sky image, and the ordinary image can be omitted, the trouble of classifying and managing the captured ordinary images is reduced.

[0147] In the above-described embodiment, the created content is described by taking an all-sky image content including an all-sky image as a wide-angle image as an example. However, the wide-angle image is not limited to the above-described all-sky image, and may be a horizontal 360-degree panoramic image, or may be an image capturing a part of the all-sky or the horizontal 360-degree panorama. Here, the wide-angle image refers to an image having a larger viewing angle than an ordinary image, and preferably has a viewing angle of at least 180 degrees or more horizontally. In addition, the wide-angle image may be a large-viewing-angle image captured by one or more photographing apparatuses. The ordinary image is a still image and a video, and is an image having a relatively narrow viewing angle compared to the wide-angle image. Specifically, the viewing angle is 100 to 25 degrees, preferably between 70 and 45 degrees.

[0148] In the description of the above embodiments, the omnidirectional image and the ordinary image are assumed to be still images. However, neither the omnidirectional image nor the ordinary image is limited to still images, and may also be an animation, a semi-animation (real-time video, interval photography image, animation) composed of a series of still images, etc. In this case, in addition to the position of the above omnidirectional image, it is further associated with time information such as time and frame interval, and the association between images is performed.

[0149] Each function of the above embodiments can be implemented by one or more processing circuits. Here, the "processing circuit" in this specification refers to a programmed processor that executes each function through software like a processor installed by a circuit, an ASIC (Application Specific Integrated Circuit) designed to execute the above functions, and devices including DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), and traditional circuit modules, etc.

[0150] The above functions can also be implemented by a computer-executable program written in traditional programming languages or object-oriented programming languages such as compilers, C, C++, C#, Java (registered trademark), etc., and can be distributed by being stored in a storage medium readable by devices such as ROM, EEPROM, EPROM, flash memory, floppy disk, CD-ROM, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, Blu-ray disc, SD card, MO, etc., or distributed through a communication circuit.

[0151] So far, one embodiment of the present invention has been described. However, the embodiments of the present invention are not limited to the above embodiments, and there may be other different embodiments, or changes within the scope obvious to those skilled in the art such as adding or deleting to the above embodiments of this article. However, no matter what kind of embodiment, as long as it achieves the functions and effects of the present invention, it belongs to the scope of the present invention.

[0152] Symbol Explanation

[0153] 10 All - sphere photography device, 12 Photographed object, 14 Housing, 18 Operation button, 20 Urban - rural optical system, 22 Image - capturing element, 50 Information terminal, 60 Internet, 70 Content - disclosure server, 112 CPU, 114 ROM, 116 Image - processing module, 118 Video - compression module, 119 Still - image compression module, 120 DRAM interface, 122 External - register interface, 124 External - sensor interface, 126 USB interface, 128 Serial module, 129 Video - output interface, 130 Photography element, 132 DRAM, 134 External register, 136 Acceleration sensor, 138 USB connector, 140 Wireless module, 152 CPU, 154 RAM, 156 HDD, 158 Input device, 160 External register, 162 Display, 164 Wireless module, 166 USB connector, 168 Built - in camera, 200 Function module, 202 Communication section, 204 Built - in camera control section, 206 Display control section, 208 Input section, 220 All - sphere image content creation section, 222 All - sphere photography device control section, 226 Built - in photography function call section, 230 Image - information management section, 232 Associated - location specification section., 234 UI section, 240 Content storage section.

Claims

1. An information processing apparatus having a communication unit and a photographing unit, wherein: an application program is installed therein, and the information processing apparatus can have the following functions by means of the application program: a first acquisition unit for controlling an external photographing apparatus to photograph a wide-angle image through the communication unit and acquiring the wide-angle image; a second acquisition unit for controlling the photographing unit to photograph a normal image and acquiring the normal image; a management unit for associating and managing the wide-angle image acquired by the first acquisition unit with the normal image acquired by the second acquisition unit; and a content production unit for producing content that can be displayed on a display device based on the associated wide-angle image and normal image, and the content can display a marker indicating the position where the normal image is associated with the wide-angle image on the wide-angle image; wherein, when a position specifying operation clicked by a user is detected in a display area for displaying a room image on the display device, position coordinates related to the specifying operation on the display area are obtained, a photographing instruction is sent to the external photographing apparatus, and then the wide-angle image photographed by the external photographing apparatus is received and associated with the position coordinates related to the specifying operation on the display area; after an instruction to add a note to the wide-angle image displayed in the application program is given, the position corresponding to the center of the display area on the wide-angle image or the position on the wide-angle image corresponding to the position clicked in the display area on the wide-angle image is determined as the associated position for adding the note, the photographing unit is instructed to photograph a normal image, and the normal image is associated with the associated position of the wide-angle image as a note.

2. The information processing apparatus according to claim 1, wherein: it further has: a position specifying unit for determining a specified position in the wide-angle image photographed by the external photographing apparatus, and a display unit and an input unit, wherein the management unit associates and manages the normal image with the specified position in the wide-angle image; the position specifying unit designates the area specified through the input unit in the display area when the wide-angle image is displayed on the display unit as the specified position in the wide-angle image; wherein the display unit embeds and displays the normal image as a note in the wide-angle image, and a specifying operation is performed on a marker provided at the note position in the wide-angle image, so that the normal image associated with the specified position pops up and is displayed on the display unit.

3. The information processing apparatus according to claim 1, wherein: the wide-angle image is an all-sky image or an image with a viewing angle of 180 degrees or more, the display area is an area representing an image obtained by projecting a part of the all-sky image or the image with a viewing angle of 180 degrees or more onto a plane, the specified position is a coordinate on the all-sky image or the image with a viewing angle of 180 degrees or more.

4. The information processing apparatus according to any one of claims 1 to 3, wherein: Furthermore, it is further provided with an external device control unit for controlling the external imaging device through the communication unit, causing the external imaging device to perform wide-angle image photography, and sending the captured wide-angle image to the information processing device.

5. The information processing device according to any one of claims 1 to 3, characterized in that it further has a first providing unit for providing a first instruction unit for receiving an instruction to cause the external imaging device to perform photography; and a second providing unit for providing, after the external imaging device performs photography, a second instruction unit for receiving an instruction for invoking the photography function of the photography unit.

6. The information processing device according to any one of claims 1 to 3, characterized in that the management unit further manages the ordinary image obtained by the second obtaining unit from the photography unit in association with the content included in the wide-angle image.

7. The information processing device according to any one of claims 1 to 3, characterized in that the management unit further manages either or both of the recorded image stored in the storage device provided in the information processing device and the externally obtained image received from the image management system communicating with the information processing device in association with the wide-angle image.

8. A computer-readable storage medium storing a program for causing an information processing device having a communication unit and a photography unit to implement the following functional units: a first obtaining unit for controlling an external imaging device to capture a wide-angle image through the communication unit and obtaining the wide-angle image; a second obtaining unit for controlling the photography unit to capture an ordinary image and obtaining the ordinary image; a management unit for managing the wide-angle image obtained by the first obtaining unit in association with the ordinary image obtained by the second obtaining unit; and a content production unit for producing content for display on a display device based on the associated wide-angle image and ordinary image, the content capable of displaying a marker indicating the position associated with the ordinary image and the wide-angle image on the wide-angle image, wherein, when a position specifying operation clicked by the user is detected in the display area of the display room image of the display device, the position coordinates related to the specifying operation on the display area are obtained, a shooting instruction is issued to the external imaging device, then the captured wide-angle image is received from the external imaging device, and the received wide-angle image is associated with the position coordinates related to the specifying operation on the display area; after an instruction to add an annotation to the wide-angle image displayed in the application program is given, the position corresponding to the center of the display area on the wide-angle image or the position on the wide-angle image corresponding to the position clicked in the display area on the wide-angle image is determined as the associated position for adding the annotation, the photography unit is instructed to capture an ordinary image, and the ordinary image is associated with the associated position of the wide-angle image as the annotation.

9. The storage medium according to claim 8, characterized in that the program further includes a position specifying unit for determining a specified position in the wide-angle image captured by the external imaging device, and a display unit and an input unit Among them, the management unit manages by associating the specified position in the normal image with that in the wide-angle image; When the position specifying unit displays the wide-angle image on the display unit, the area specified by the input unit in the display area is defined as the specified position in the wide-angle image; Among them, the display unit embeds the normal image as an annotation into the wide-angle image for display, and designates an operation on a marker provided at the annotation position in the wide-angle image, so that the normal image associated with the specified position pops up and is displayed on the display unit.

10. The storage medium according to claim 8 or 9, characterized in that the program further includes the following functional units a first providing unit for providing a first indicating unit for receiving an instruction to cause the external photographing device to perform photographing; and a second providing unit for providing a second indicating unit for receiving an instruction to call the photographing function of the photographing unit after the external photographing device performs photographing.

11. A computer device having a communication unit and a photographing unit, which has a processor and a storage unit storing a program. By executing the program by the processor, the computer device implements the following functional units a first obtaining unit for controlling an external photographing device to photograph a wide-angle image through the communication unit and obtaining the wide-angle image; a second obtaining unit for controlling the photographing unit to photograph a normal image and obtaining the normal image; a management unit for managing by associating the wide-angle image obtained by the first obtaining unit with the normal image obtained by the second obtaining unit; and a content production unit for producing content for display on a display device according to the associated wide-angle image and normal image, and the content can display a marker indicating the position associated with the normal image and the wide-angle image on the wide-angle image. Among them, when a position specifying operation clicked by a user is detected in the display area of the display room image of the display device, the position coordinates related to the specifying operation on the display area are obtained, a photographing instruction is sent to the external photographing device, and then the wide-angle image taken is received from the external photographing device, and the received wide-angle image is associated with the position coordinates related to the specifying operation on the display area; after an instruction to add an annotation to the wide-angle image displayed in the application program is given, the position corresponding to the center of the display area on the wide-angle image, or the position on the wide-angle image corresponding to the position clicked in the display area on the wide-angle image, is determined as the associated position for adding the annotation, the photographing unit is instructed to photograph a normal image, and the normal image is associated with the associated position of the wide-angle image as an annotation.

12. An information processing method for an information processing device having a communication unit and a photographing unit, characterized in that it includes the following steps a first obtaining step of controlling an external photographing device to photograph a wide-angle image through the communication unit and obtaining the wide-angle image; a second obtaining step of controlling the photographing unit to photograph a normal image and obtaining the normal image; A management step of managing the wide-angle image obtained in the first acquisition step and the normal image obtained in the second acquisition step in an associated manner; and A content production step of producing content for display on a display device based on the associated wide-angle image and normal image, the content being capable of displaying a marker indicating the position where the normal image is associated with the wide-angle image on the wide-angle image, wherein when a position specifying operation clicked by a user is detected in the display area of the display room image of the display device, position coordinates related to the specifying operation on the display area are obtained, a shooting instruction is sent to an external imaging device, and then the wide-angle image taken by the external imaging device is received and associated with the position coordinates related to the specifying operation on the display area; after an instruction to add a comment to the wide-angle image displayed in the application is given, the position corresponding to the center of the display area on the wide-angle image or the position on the wide-angle image corresponding to the position clicked in the display area on the wide-angle image is determined as the associated position for adding the comment, the imaging unit is instructed to shoot a normal image, and the normal image is associated as a comment with the associated position of the wide-angle image.

Images