PROGRAM, INFORMATION PROCESSING APPARATUS AND INFORMATION PROCESSING METHOD

Abstract

To provide a program or the like capable of displaying content, corresponding to an earth condition that changes from moment to moment, superimposed on a photographic image.SOLUTION: A computer downloads time series data on a cloud distribution image showing a cloud distribution of the world by equirectangular projection, downloads time series data on a rainfall distribution image showing a rainfall distribution of the world by the equirectangular projection, and downloads time series data on an air temperature distribution image showing an air temperature distribution of the world by the equirectangular projection. The computer selectably displays a cloud image icon, a rainfall image icon or an air temperature image icon for selecting any one of the cloud distribution image, the rainfall distribution image or the air temperature distribution image. The computer acquires a photographic image of a globe from a camera, and continuously reproduces a corrected cloud distribution image that continues in time series while correcting each cloud distribution image so as to cover the entire globe in the acquired photographic image and match an image drawn on the globe.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a program, an information processing device, and an information processing method. [Background technology]

[0002] In recent years, products using an AR (Augmented Reality) system have been developed. In the AR system, an application program for recognizing an AR marker is installed in an information device having a camera and a display unit, such as a smartphone, a tablet terminal, a portable game machine, etc. When an object with an AR marker is photographed using such an information device, the information device recognizes the AR marker and displays content corresponding to the recognized AR marker superimposed on the photographed image. This allows a user to view an image on the display unit in which the content is superimposed on the object being photographed. Patent Document 1 discloses a device that photographs a globe and displays a virtual scene superimposed on the globe in the photographed image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2017-522682 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional techniques have a problem in that the content that is superimposed on a captured image of a subject, such as a globe, is limited to pre-registered content.

[0005] The present disclosure has been made in light of the above circumstances, and its purpose is to provide a program or the like that is capable of overlaying content that corresponds to the ever-changing state of the Earth onto a captured image. [Means for solving the problem]

[0006] A program according to one aspect of the present disclosure causes a computer to execute a process of downloading time series data of a cloud distribution image showing cloud distribution around the world in equirectangular projection, downloading time series data of a rainfall distribution image showing rainfall distribution around the world in equirectangular projection, downloading time series data of a temperature distribution image showing temperature distribution around the world in equirectangular projection, selectably displaying a cloud image icon, a rainfall image icon, or a temperature image icon for selecting one of the cloud distribution image, the rainfall distribution image, or the temperature distribution image, acquiring a photographed image of a globe from a camera, and when the cloud image icon is selected, correcting each cloud distribution image in equirectangular projection so that it covers the entire globe in the acquired photographed image and matches the image drawn on the globe, while continuously playing back chronologically consecutive corrected cloud distribution images. Effect of the Invention

[0007] According to one aspect of the present disclosure, content that corresponds to the ever-changing state of the Earth can be used as content to be displayed overlaid on a captured image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing a configuration example of an AR system. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of a server and a user terminal. [Diagram 3] FIG. 2 is a schematic diagram showing an example of the configuration of a weather content DB. [Figure 4] 13 is a flowchart showing an example of a procedure for a weather content acquisition process. [Diagram 5] FIG. 13 is a schematic diagram showing an example of a cloud distribution image. [Figure 6] 13 is a flowchart illustrating an example of a procedure for displaying an AR image. [Figure 7] 13 is a flowchart illustrating an example of a procedure for displaying an AR image. [Figure 8] FIG. 13 is a schematic diagram showing a display example of an AR image. [Figure 9] FIG. 1 is a schematic diagram for explaining an AR image. [Figure 10] FIG. 13 is a schematic diagram showing a display example of an AR image. [Figure 11] 13 is a flowchart showing an example of a display processing procedure for an AR image using a border line image. [Figure 12] FIG. 13 is a schematic diagram showing a display example of an AR image. [Figure 13] 13 is a flowchart showing an example of an AR image switching process procedure. [Figure 14] FIG. 13 is a schematic diagram showing a display example of an AR image. [Figure 15] 11A and 11B are schematic diagrams showing other configuration examples of the earth ball. [Figure 16] 13 is a flowchart showing an example of a display processing procedure of an AR image according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, a program, an information processing device, and an information processing method according to the present disclosure will be described in detail with reference to the drawings showing embodiments thereof.

[0010] (Embodiment 1) An AR system for photographing an Earth ball (globe), which is a sphere having an AR marker on its surface, as a subject will be described. FIG. 1 is a schematic diagram showing a configuration example of an AR system. The AR system of this embodiment includes a server 10, a user terminal 20, and an Earth ball 30. The server 10 and the user terminal 20 can be connected to a network N such as the Internet, and transmit and receive information via the network N. The AR system of this embodiment also includes a web server 40 (hereinafter, referred to as the weather server 40) that publishes weather information via the network N, and the server 10 acquires weather information from the weather server 40 via the network N. The weather information may be, for example, a cloud distribution image showing the cloud conditions on the earth provided by Custom Weather, Inc., a temperature distribution image showing the temperatures in various places on the earth, and a rainfall distribution image (Satellite Global Precipitation Map: GSMaP) showing the precipitation conditions on the earth provided by, for example, the JAXA Earth Observation Research Center (EORC). Therefore, the weather server 40 includes a web server managed by, for example, Custom Weather, Inc. and the JAXA Earth Observation Research Center. The weather information and weather server 40 is not limited to these.

[0011] The server 10 is an information processing device capable of various information processing and transmitting and receiving information, such as a server computer or a personal computer. The server 10 may be provided in a plurality of units, may be realized by a plurality of virtual machines provided in one server device, or may be realized by using a cloud server. The server 10 performs various information processing, such as a process of acquiring weather information from the weather server 40, a process of generating weather information to be provided from the acquired weather information, and a process of providing weather information in response to a request from the user terminal 20. The weather server 40 is a general web server, so a detailed description will be omitted. The weather server 40 stores data to be disclosed (weather information), and when a request is received from another device via the network N, the weather server 40 performs a process of providing the requested data via the network N.

[0012] The user terminal 20 is a terminal device of a user who photographs the earth ball 30 and uses the AR system. The user terminal 20 is an information processing device capable of various information processing and transmitting and receiving information, such as a smartphone, a mobile phone, a tablet terminal, a personal computer, a game machine, etc. If the user terminal 20 is a portable device, it is easy to operate. The user terminal 20 performs various information processing, such as a process of acquiring weather information from the server 10, a process of photographing the earth ball 30, and a process of displaying the weather information by superimposing it on the photographed image.

[0013] The Earth Ball 30 is, for example, a resin globe, but is not limited thereto. A map of the Earth is printed (drawn) on the surface of the Earth Ball 30. The map printed on the Earth Ball 30 includes a topographical continental map 31 (topographical map) in which land is represented by a color according to the altitude, and an oceanic map 32 (maritime chart) in which ocean is represented by a color according to the depth of the water. Thus, in the Earth Ball 30, rivers, lakes, plains, mountains, deserts, etc. are colored according to the topography, and the oceans in the map are colored according to the depth of the water. As shown in FIG. 1, the Earth Ball 30 does not have the names of the continents and oceans in the map, national borders, the names of countries, the names of cities, etc. printed on it. Meanwhile, the Earth Ball 30 has a cloud-shaped cloud mark 33 (dummy cloud image) printed at an appropriate position on the oceanic map 32. The cloud mark 33 is configured to have a shape imitating a cirrus cloud (cirrus cloud), a cirrocumulus cloud (scale cloud, sardine cloud), an altocumulus cloud (sheep cloud), etc., but is not limited to these. In the Earth ball 30 configured in this way, the continental map 31 and the oceanic map 32, and the cloud mark 33 drawn on the oceanic map 32 are used as AR markers.

[0014] 2 is a block diagram showing an example of the configuration of the server 10 and the user terminal 20. The server 10 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, etc., and these units are connected to each other via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), or a GPU (Graphics Processing Unit). The control unit 11 appropriately executes a control program 12P stored in the storage unit 12 to perform various information processing and control processing performed by the server 10.

[0015] The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), etc. The storage unit 12 prestores a control program 12P executed by the control unit 11 and various data required for executing the control program 12P. The storage unit 12 also temporarily stores data generated when the control unit 11 executes the control program 12P. The storage unit 12 also stores a weather content DB (database) 12a, which will be described later. The weather content DB 12a may be stored in another storage device connected to the server 10, or in another storage device with which the server 10 can communicate.

[0016] The communication unit 13 is an interface for connecting to the network N by wired or wireless communication, and transmits and receives information to and from other devices via the network N. The input unit 14 accepts operation input by a user who operates the server 10, and sends a control signal corresponding to the operation content to the control unit 11. The display unit 15 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control unit 11. The input unit 14 and the display unit 15 may be a touch panel configured as an integrated unit.

[0017] The reading unit 16 reads information stored in a portable storage medium 1a including a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc)-ROM, a USB (Universal Serial Bus) memory, an SD (Secure Digital) card, etc. The control program 12P and various data stored in the storage unit 12 may be read by the control unit 11 from the portable storage medium 1a via the reading unit 16 and stored in the storage unit 12. The control program 12P and various data stored in the storage unit 12 may be downloaded by the control unit 11 from another device via the communication unit 13 and stored in the storage unit 12.

[0018] The user terminal 20 includes a control unit 21, a storage unit 22, a communication unit 23, an input unit 24, a display unit 25, a camera 26, etc., and these units are connected to each other via a bus. The control unit 21, the storage unit 22, the communication unit 23, the input unit 24, and the display unit 25 of the user terminal 20 are configured similarly to the control unit 11, the storage unit 12, the communication unit 13, the input unit 14, and the display unit 15 of the server 10, and therefore detailed explanations are omitted. In addition to the control program 22P executed by the control unit 21, the storage unit 22 of the user terminal 20 stores an AR application program 22AP (hereinafter referred to as an AR application 22AP) for displaying an AR image superimposed on the earth ball 30 in the captured image when the earth ball 30 is captured. The AR application 22AP may be downloaded by the control unit 21 from another device via the communication unit 23 and stored (installed) in the storage unit 22. In addition, if the user terminal 20 has a reading unit that reads information stored in a portable storage medium, the AR application 22AP may be read by the control unit 21 from the portable storage medium via the reading unit and stored (installed) in the storage unit 22.

[0019] Camera 26 is an imaging device that takes pictures according to instructions from control unit 21, and transmits the acquired captured image (image data) to storage unit 22 for storage. Camera 26 may be built into user terminal 20, or may be externally attached to user terminal 20. When camera 26 is externally attached, user terminal 20 has a connection unit capable of connecting an external camera, or a wireless communication unit capable of wireless communication with an external camera, and acquires image data captured by the external camera via the connection unit or wireless communication unit.

[0020] The user terminal 20 may be an HMD (Head Mounted Display) type information device that uses a goggle-type or eyeglass-type head mounted display for the display unit 25. In this case, it is desirable to use a microphone for voice input, or a gaze input device for detecting the user's gaze, for the input unit 24. The user terminal 20 may be any device as long as it is an information device having a camera 26 and a display unit 25.

[0021] FIG. 3 is a schematic diagram showing an example of the configuration of the weather content DB 12a. The weather content DB 12a of this embodiment stores various weather contents (weather images) to be displayed by being superimposed on the captured image of the Earth ball 30. In the AR system of this embodiment, as the weather contents to be displayed by being superimposed on the captured image of the Earth ball 30, a cloud distribution image showing the cloud distribution in the world (on the earth), a rainfall distribution image showing the rainfall distribution (precipitation distribution) in the world, and a temperature distribution image showing the temperature distribution in the world are used. Therefore, the weather content DB 12a includes a cloud distribution image DB that stores the cloud distribution image, a rainfall distribution image DB that stores the rainfall distribution image, and a temperature distribution image DB that stores the temperature distribution image. Note that the weather content is not limited to the cloud distribution image, the rainfall distribution image, and the temperature distribution image, and may include image content related to other weather information. In this embodiment, an example is shown in which the cloud distribution image DB, the rainfall distribution image DB, and the temperature distribution image DB are each configured as a separate DB, but the cloud distribution image, the rainfall distribution image, and the temperature distribution image may be stored in one DB.

[0022] The cloud distribution image DB shown in FIG. 3A includes an image ID column, a date and time column, a cloud distribution image column, etc., and stores the date and time and the data of the cloud distribution image in association with the image ID. The image ID column stores identification information (image ID) assigned to the cloud distribution image. The date and time column and the cloud distribution image column store the date and time and the data of the cloud distribution image at each date and time, respectively. The data of the cloud distribution image is stored in a predetermined area of ​​the storage unit 12 or in another storage device, and the cloud distribution image column stores information for reading out the data of the cloud distribution image (for example, a file name indicating the storage location of the data). The image ID stored in the cloud distribution image DB is issued and stored by the control unit 11 when the control unit 11 acquires new data of a cloud distribution image from the weather server 40. The date and time and the data of the cloud distribution image stored in the cloud distribution image DB are stored by the control unit 11 when the control unit 11 acquires them from the weather server 40. The control unit 11 acquires information on the date and time when each cloud distribution image was collected from the weather server 40, together with the data of the cloud distribution image, from the weather server 40. The contents stored in the cloud distribution image DB are not limited to the example shown in FIG. 3A, and various types of information relating to cloud distribution and cloud distribution images can be stored.

[0023] The rainfall distribution image DB and the temperature distribution image DB have the same configuration as the cloud distribution image DB. Specifically, the rainfall distribution image DB shown in FIG. 3B includes an image ID string, a date and time string, a rainfall distribution image string, etc., and stores the date and time and the rainfall distribution image data in association with the image ID. The temperature distribution image DB shown in FIG. 3C includes an image ID string, a date and time string, a temperature distribution image string, etc., and stores the date and time and the temperature distribution image data in association with the image ID. As with the cloud distribution image, the control unit 11 acquires the rainfall distribution image and the temperature distribution image from the weather server 40 together with the data of each distribution image. The storage contents of the rainfall distribution image DB are not limited to the example shown in FIG. 3B, and the storage contents of the temperature distribution image DB are not limited to the example shown in FIG. 3C.

[0024] The following describes the process in which the server 10 acquires (downloads) weather content (cloud distribution image, rainfall distribution image, and temperature distribution image) from the weather server 40 in the AR system of this embodiment. FIG. 4 is a flowchart showing an example of the process procedure for acquiring weather content, and FIG. 5 is a schematic diagram showing an example of a cloud distribution image. In FIG. 4, the left side shows the process performed by the server 10, and the right side shows the process performed by the weather server 40. The following process is executed by the control unit 11 in accordance with a control program 12P stored in the memory unit 12 of the server 10. Part of the following process may be realized by a dedicated hardware circuit.

[0025] In the AR system of this embodiment, the weather server 40 periodically (e.g., every hour) adds weather contents to be provided (cloud distribution image, rainfall distribution image, and temperature distribution image). The control unit 11 of the server 10 periodically (e.g., every hour) downloads the weather contents (cloud distribution image, rainfall distribution image, and temperature distribution image) from the weather server 40. This allows the server 10 to sequentially download the weather contents provided by the weather server 40 and store them in the storage unit 12. Note that the weather contents of the cloud distribution image, rainfall distribution image, and temperature distribution image may be downloaded from different weather servers 40 or may be downloaded from the same weather server 40.

[0026] The control unit 11 of the server 10 judges whether it is time to download any of the weather contents (cloud distribution image, rainfall distribution image, or temperature distribution image) from the weather server 40 (S11). The server 10 stores in advance in the storage unit 12 the download timing for each weather content, and the control unit 11 judges whether the download timing stored in the storage unit 12 for each weather content has arrived. For example, if downloading is to be performed once a day, a predetermined time within the day is set as the download timing, and if downloading is to be performed a predetermined number of times a day, a predetermined number of times within the day are set as the download timing.

[0027] If it is determined that the download timing has not arrived (S11: NO), the control unit 11 waits while performing other processes until the download timing arrives. If it is determined that the download timing has arrived (S11: YES), the control unit 11 requests the latest data of the weather content to be downloaded from the weather server 40 that provides this weather content (S12). The control unit 11 may compare each weather content already stored in the weather content DB 12a with the weather content that the weather server 40 can provide, and request the weather server 40 for the weather content that has not been downloaded.

[0028] The weather server 40 reads out the weather content requested by the server 10 from a storage unit (not shown) (S13) and transmits it to the server 10 via the network N (S14). Specifically, when a cloud distribution image is requested, the weather server 40 transmits the latest data of the cloud distribution image to the server 10, when a rainfall distribution image is requested, the latest data of the rainfall distribution image to the server 10, and when a temperature distribution image is requested, the latest data of the temperature distribution image to the server 10. FIG. 5 shows an example of a cloud distribution image, and the cloud distribution image is an image in which an image showing cloud distribution is superimposed on a world map represented by, for example, equirectangular projection. In the example shown in FIG. 5, the hatched areas represent clouds. Similarly, the rainfall distribution image is an image in which an image showing rainfall distribution is superimposed on a world map represented by equirectangular projection, and the temperature distribution image is an image in which an image showing temperature distribution is superimposed on a world map represented by equirectangular projection.

[0029] The weather content provided by the weather server 40 is data of weather images showing each weather information associated with the date and time at which the data was acquired. Therefore, the control unit 11 of the server 10 can acquire date and time information and data of weather images acquired at each date and time by acquiring weather content from the weather server 40. Therefore, when the control unit 11 acquires weather content from the weather server 40, it issues an image ID and stores the date and time included in the weather content and the weather image data in the weather content DB 12a in association with the issued image ID (S15). Specifically, when the control unit 11 receives a cloud distribution image as the weather content, it stores the date and time and the data of the cloud distribution image in association with the image ID in the cloud distribution image DB. Similarly, when the control unit 11 receives a rainfall distribution image, it stores the date and time and the data of the rainfall distribution image in association with the image ID in the rainfall distribution image DB, and when the control unit 11 receives a temperature distribution image, it stores the date and time and the data of the temperature distribution image in association with the image ID in the temperature distribution image DB.

[0030] Through the above-mentioned process, the server 10 can sequentially acquire weather content periodically updated by the weather server 40 and store it in the weather content DB 12a. Specifically, the server 10 can download time-series data of weather images (cloud distribution image, rainfall distribution image, and temperature distribution image), and store each weather image in the weather content DB 12a with date and time information added. The weather content thus collected by the server 10 from the weather server 40 is used for an AR image that is displayed superimposed on the captured image of the earth ball 30 when the earth ball 30 is photographed using the user terminal 20.

[0031] In the AR system of this embodiment, when a user photographs the earth ball 30 using the user terminal 20, a process of displaying an AR image using weather content superimposed on the photographed image of the earth ball 30 will be described below. FIGS. 6 and 7 are flowcharts showing an example of a display process procedure for an AR image, FIGS. 8 and 10 are schematic diagrams showing a display example of an AR image, and FIG. 9 is a schematic diagram for explaining an AR image. In FIGS. 6 and 7, the left side shows the process performed by the user terminal 20, and the right side shows the process performed by the server 10. The following process is executed by the control unit 21 according to the control program 22P and the AR application 22AP stored in the storage unit 22 of the user terminal 20, and is executed by the control unit 11 according to the control program 12P stored in the storage unit 12 of the server 10. A part of the following process may be realized by a dedicated hardware circuit.

[0032] In the AR system of the present embodiment, when a user wants to see a weather image such as a cloud distribution image, a rainfall distribution image, or a temperature distribution image on the earth, the user performs a predetermined operation on the user terminal 20 to instruct the AR application 22AP to be started. When the control unit 21 of the user terminal 20 receives an instruction (start command) to start the AR application 22AP via the input unit 24, the control unit 21 starts the AR application 22AP (S21). When the control unit 21 starts the AR application 22AP, the control unit 21 starts shooting with the camera 26 (S22). When the camera 26 starts shooting, it acquires a captured image at a predetermined timing (for example, every 1 / 30 seconds or every 1 / 15 seconds). The captured images acquired by the camera 26 are sequentially sent to the display unit 25 and displayed on the display unit 25. The left side of FIG. 8 shows an example of a display screen of a captured image of the earth ball 30. In this way, the user can check the captured image of the earth ball 30 on the display unit 25 while shooting the earth ball 30 with the camera 26. The captured image may be stored in the storage unit 22 and then sent to the display unit 25 for display.

[0033] Furthermore, when the control unit 21 activates the AR application 22AP, it requests the server 10 for a cloud distribution image (first weather image showing first weather information) (S23). Specifically, the control unit 21 transmits to the server 10 a start-up signal indicating that the AR application 22AP has been activated, or a download request signal requesting download of the cloud distribution image. The control unit 21 may perform the process of step S22 after the process of step S23. When the control unit 11 of the server 10 receives the start-up signal or the download request signal of the cloud distribution image from the user terminal 20, it refers to the date and time stored in the cloud distribution image DB and reads out data of the cloud distribution image for the past 24 hours starting from the current date and time from the cloud distribution image DB (S24). Specifically, the control unit 11 extracts the date and time included in the past 24 hours from the current date and time from the date and time stored in the cloud distribution image DB, and reads out data of the cloud distribution image corresponding to the extracted date and time together with the date and time of each data. The control unit 11 may read out one cloud distribution image data every hour, one cloud distribution image data every 30 minutes or one cloud distribution image data every two hours as the cloud distribution image data for 24 hours, but is not limited to these. At this time, the control unit 11 does not read out the rainfall distribution image data and the temperature distribution image data from the rainfall distribution image DB or the temperature distribution image DB, but reads out only the cloud distribution image data from the cloud distribution image DB.

[0034] The control unit 11 transmits the data of the cloud distribution image for 24 hours read from the cloud distribution image DB to the user terminal 20 via the network N (S25). At this time, the control unit 11 gives a file name including the date and time of each data to each of the data of the cloud distribution image read from the cloud distribution image DB. For example, the control unit 11 gives the file name "cloud_2020-08-01-00.jpg" to a cloud distribution image dated at 0:00 on August 1, 2020. This allows the server 10 to generate time-series data (moving images) of the cloud distribution image by continuously combining the data of the cloud distribution image for 24 hours in the order of the file names (specifically, in the order of the date and time included in the file names). Therefore, the control unit 21 of the user terminal 20 can acquire (download) the time-series data of the cloud distribution image (first weather image) represented in equirectangular projection from the server 10 as shown in FIG. 5. The control unit 21 of the user terminal 20 stores the time series data of the cloud distribution image (24 hours' worth of cloud distribution image data) acquired from the server 10 in the storage unit 22 (S26). In this way, the user terminal 20 of this embodiment downloads only the data of the cloud distribution image (first weather image) from the server 10 without downloading the data of the rainfall distribution image (second weather image) and the data of the temperature distribution image (third weather image) immediately after starting the AR application 22AP. In addition, the user terminal 20 is configured to acquire the data of the cloud distribution image for 24 hours from the server 10 at this time, but is not limited to such a configuration. For example, the user terminal 20 may acquire the data of the cloud distribution image for a predetermined period other than 24 hours from the server 10 immediately after starting the AR application 22AP, or may acquire the data of the weather image other than the cloud distribution image for a predetermined period from the server 10. In addition, the user terminal 20 acquires the data of the cloud distribution image for 24 hours (time series data of weather information) all at once, but is not limited to such a configuration.

[0035] The control unit 21 acquires a photographed image of the earth ball 30 captured by the camera 26 (S27). The control unit 21 then recognizes an AR marker in the acquired photographed image of the earth ball 30, and specifies a photographing range (photographing area) of the earth ball 30 based on the recognized AR marker (S28). Each AR marker printed on the surface of the earth ball 30 is associated with position information (longitude and latitude information) on the earth, and the corresponding information of each AR marker and the position information is set in the AR application 22AP. Therefore, the control unit 21 can specify the position information (longitude and latitude information) associated with the AR marker recognized in the photographed image of the earth ball 30, and specify the photographing range of the earth ball 30 based on the specification result. For example, the control unit 21 specifies the photographing range of the earth ball 30 by specifying the position information (longitude and latitude information) corresponding to the image center (center position) of the photographed image of the earth ball 30 and the direction corresponding to the upward direction of the photographed image. In the photographed screen of the earth ball 30 shown on the left side of FIG. 8, the control unit 21 specifies, for example, the position information of the image center P and that the direction corresponding to the upward direction of the photographed image is north.

[0036] In the process of identifying the position information corresponding to the image center of the captured image of the Earth ball 30, for example, the control unit 21 identifies the center position (pixel at the center position) of the captured image of the circular Earth ball 30, and identifies the AR marker at the identified center position. Then, the control unit 21 can identify the position information corresponding to the image center by reading out the position information associated with the identified AR marker. If there is no AR marker at the identified center position, the control unit 21 identifies a predetermined number (for example, two) of AR markers in the vicinity of the center position. In this case, the control unit 21 calculates the position information corresponding to the image center based on the positional relationship in the image between the identified predetermined number of AR markers and the center position, and the position information (longitude and latitude information) of each AR marker. For example, when two AR markers are used, the control unit 21 calculates the amount of change in longitude and latitude corresponding to one pixel in the left-right direction and up-down direction in the image based on the positions of the two AR markers in the image and the longitudes and latitudes associated with the two AR markers. Then, the control unit 21 calculates position information (longitude and latitude) corresponding to the center position (image center) based on the position in the image, longitude, and latitude of one of the AR markers, and the center position of the earth ball 30. The method of calculating the position information corresponding to the image center of the captured image of the earth ball 30 is not limited to the method using two AR markers. For example, if a direction corresponding to the upward direction of the captured image is specified, position information corresponding to the image center of the captured image of the earth ball 30 can be specified using one AR marker. In addition, position information corresponding to the image center of the captured image of the earth ball 30 may be calculated using three or more AR markers. In addition, in the process of specifying the direction corresponding to the upward direction of the captured image of the earth ball 30, for example, the control unit 21 can specify any two AR markers and specify the direction corresponding to the upward direction of the captured image based on the position information (longitude and latitude) of the specified AR markers.

[0037] The control unit 21 generates an AR image to be displayed superimposed on the captured image of the earth ball 30 based on the data of the cloud distribution image acquired from the server 10 and the shooting range of the earth ball 30 specified in step S28 (S29). First, the control unit 21 extracts an area corresponding to the specified shooting range of the earth ball 30 from the data of the cloud distribution image. Specifically, the control unit 21 specifies a hemispherical area having the position information (longitude and latitude) of the specified image center P as the center position, and extracts an image included in the specified hemispherical area from the cloud distribution image. Then, the control unit 21 transforms the extracted cloud distribution image of the hemispherical area into a circular image to display it in a circular shape on the captured image of the earth ball 30. As a result, the control unit 21 can generate (correct) a circular cloud distribution image by extracting a part of the cloud distribution image from a rectangular cloud distribution image expressed by equirectangular projection so as to match the image (map) drawn on the earth ball 30 in the captured image, as shown in FIG. 9. The control unit 21 generates time-series data (moving image data) of circular cloud distribution images by generating circular cloud distribution images for each of the 24-hour cloud distribution image data acquired from the server 10. Note that, for example, when there are areas on the earth where there is no cloud distribution image data, the control unit 21 may generate cloud distribution images by adding dummy cloud images to such areas.

[0038] Then, the control unit 21 displays the generated AR image superimposed on the photographed image of the earth ball 30 being displayed on the display unit 25 (S30). As a result, as shown on the right side of FIG. 8, the AR image of the cloud distribution image processed into a circle can be superimposed on the photographed image of the earth ball 30. The control unit 21 may generate the AR image to be superimposed on the photographed image of the earth ball 30 from the cloud distribution image by using, for example, a rendering function (texture mapping) by computer graphics. In this case, the control unit 21 specifies the photographed range of the earth ball 30 specified in step S28 as an extraction area from the data of the cloud distribution image to the rendering engine. As a result, the rendering engine maps each pixel of the cloud distribution image to each pixel of the photographed image of the earth ball 30, and as a result, the AR image of the cloud distribution image is superimposed on the photographed image of the earth ball 30. When performing such processing, the control unit 21 executes the processing of steps S29 to S30 collectively.

[0039] The control unit 21 may display one cloud distribution image (for example, the first cloud distribution image) of the time series data of the generated circular cloud distribution image in an overlapping manner, or may sequentially switch between the cloud distribution images to continuously display cloud distribution images for 24 hours in a frame-by-frame manner. When the control unit 21 displays the latest cloud distribution image in an overlapping manner on the image captured by the Earth ball 30, the user can see the cloud distribution on the Earth in real time. The control unit 21 may also display the date and time corresponding to the cloud distribution image displayed in an overlapping manner on the image captured by the Earth ball 30 in association with the AR image, as shown in the screen on the right side of FIG. 8. By the above-mentioned process, the control unit 21 can continuously play back the corrected cloud distribution images that are consecutive in time series while correcting each cloud distribution image of the equirectangular projection so that it covers the entire Earth ball 30 in the captured image and matches the captured image of the Earth ball 30 (i.e., a map within the capture range of the Earth ball 30, etc.).

[0040] The control unit 21 judges whether the shooting range of the earth ball 30 has been changed based on the captured images of the earth ball 30 captured sequentially by the camera 26 (S31). For example, the control unit 21 calculates the difference between the captured images of the earth ball 30 captured sequentially via the camera 26, and judges whether the shooting range of the earth ball 30 has been changed based on the calculated difference. If it is judged that the shooting range of the earth ball 30 has been changed (S31: YES), the control unit 21 returns to the process of step S27 and acquires a captured image of the earth ball 30 by the camera 26 for the changed shooting range (S27). Then, the control unit 21 performs the processes of steps S28 to S30 based on the acquired captured images, thereby generating an AR image of a cloud distribution image corresponding to the changed shooting range of the earth ball 30, and can display it superimposed on the captured image of the earth ball 30. The control unit 21 repeats the processes of steps S27 to S30 every time the shooting range of the earth ball 30 is changed. This allows the center position of the captured image of the earth ball 30 to be identified in real time according to the appropriately changed shooting range of the earth ball 30, and an AR image can be generated from the cloud distribution image based on the identified center position (shooting range of the earth ball 30). Therefore, the AR image of the cloud distribution image at a location (location information) included in the appropriately changed shooting range can be displayed superimposed on the captured image of the earth ball 30. Therefore, when the user turns the earth ball 30 in his / her hand or changes the shooting direction of the earth ball 30 by the camera 26, the shooting range of the earth ball 30 is changed, and the AR image (weather content) superimposed on the captured image of the earth ball 30 is updated in real time according to the changed shooting range.

[0041] In the screen shown on the right side of FIG. 8, an image switching button B1 is displayed for selecting one of a cloud distribution image, a rainfall distribution image, and a temperature distribution image as an AR image to be displayed superimposed on the captured image of the earth ball 30. The image switching button B1 has a cloud image icon for selecting a cloud distribution image (first weather image), a rainfall image icon for selecting a rainfall distribution image (second weather image), and a temperature image icon for selecting a temperature distribution image (third weather image). When the user performs a touch operation on the image switching button B1 in a state where the cloud image icon is displayed in the foreground as shown in the screen on the left side of FIG. 10, the image switching button B1 is in a state where the rainfall image icon is displayed in the foreground as shown in the screen on the right side of FIG. 10, when the user performs a touch operation on the image switching button B1 in a state where the rainfall image icon is displayed in the foreground as shown in the screen on the right side of FIG. 10, the image switching button B1 is in a state where the temperature image icon is displayed in the foreground as shown in the screen on the bottom right of FIG. In this way, the image switching button B1 can issue a switching instruction to switch the image to be displayed in the order of a cloud distribution image, a rainfall distribution image, and a temperature distribution image each time the user performs a touch operation.

[0042] Therefore, when the control unit 21 determines that the shooting range has not been changed (S31: NO), it determines whether or not, for example, the image switching button B1 has been operated (S32). When it determines that the image switching button B1 has been operated (S32: YES), the control unit 21 requests the server 10 for a rainfall distribution image (second weather image) or a temperature distribution image (third weather image) depending on the number of times it has been operated (S33). Note that when the image switching button B1 is operated once with a cloud image icon displayed in the foreground, the control unit 21 requests a rainfall distribution image, and when it is operated twice, the control unit 21 requests a temperature distribution image. When it determines that the image switching button B1 has not been operated (S32: NO), the control unit 21 proceeds to the process of step S42.

[0043] When the user terminal 20 requests a rainfall distribution image or a temperature distribution image, the control unit 11 of the server 10 reads out the rainfall distribution image or temperature distribution image data for the most recent 24 hours from the rainfall distribution image DB or the temperature distribution image DB (S34). Specifically, the control unit 11 refers to the date and time stored in the rainfall distribution image DB or the temperature distribution image DB, and reads out the rainfall distribution image or temperature distribution image data corresponding to the date and time included in the past 24 hours from the current date and time. The control unit 11 transmits the read out rainfall distribution image or temperature distribution image data for the 24 hours to the user terminal 20 (S35). At this time, the control unit 11 also gives a file name including the date and time of each data to each of the read out rainfall distribution image data, and gives a file name including the date and time of each data to each of the temperature distribution image data. In this way, the server 10 can generate time series data (moving images) of the rainfall distribution image or the temperature distribution image by continuously combining the rainfall distribution image or temperature distribution image data for the 24 hours. The rainfall distribution image and the temperature distribution image are expressed using the equirectangular projection, similar to the cloud distribution image shown in FIG.

[0044] The control unit 21 of the user terminal 20 acquires the 24-hour rainfall distribution image or temperature distribution image data (time-series data of the rainfall distribution image or temperature distribution image) transmitted by the server 10, and stores it in the storage unit 22 (S36). Then, the control unit 21 performs the same process as steps S27 to S28, acquires a photographed image of the Earth ball 30 by the camera 26 (S37), and identifies the photographing range of the Earth ball 30 based on the AR marker in the photographed image of the Earth ball 30 (S38).

[0045] Next, the control unit 21 generates an AR image to be displayed superimposed on the captured image of the Earth ball 30 based on the rainfall distribution image or temperature distribution image data acquired from the server 10 and the captured range of the Earth ball 30 specified in step S38 (S39). The processing here is the same as step S29, except that the processing target is changed from the cloud distribution image data to the rainfall distribution image or temperature distribution image data. As a result, for the rainfall distribution image or temperature distribution image data, a circular distribution image is generated by extracting a part of the distribution image from the rectangular distribution image represented by the equirectangular projection, as shown in FIG. 9.

[0046] Then, the control unit 21 displays the generated AR image superimposed on the photographed image of the earth ball 30 being displayed on the display unit 25 (S40). As a result, when the image switching button B1 is operated once while a cloud distribution image (AR image) is displayed as in the screen on the left side of FIG. 10, the AR image displayed superimposed on the photographed image of the earth ball 30 is switched to a rainfall distribution image as in the screen on the upper right of FIG. 10. When the image switching button B1 is operated twice while a cloud distribution image (AR image) is displayed as in the screen on the left side of FIG. 10, or when the image switching button B1 is operated once while a rainfall distribution image (AR image) is displayed as in the screen on the upper right of FIG. 10, the AR image is switched to a temperature distribution image as in the screen on the lower right of FIG. 10. Note that, although the screen on the lower right of FIG. 10 displays a temperature distribution in which each temperature is hatched differently, the temperature distribution may be a heat map in which each temperature is displayed in a different color.

[0047] Even when the AR image of the rainfall distribution image or the temperature distribution image is displayed superimposed on the photographed image of the earth ball 30, the control unit 21 judges whether the photographed range of the earth ball 30 has been changed based on the photographed images of the earth ball 30 sequentially photographed by the camera 26 (S41). Then, when it is judged that the photographed range of the earth ball 30 has been changed (S41: YES), the control unit 21 returns to the process of step S37 and performs the processes of steps S37 to S40 for the photographed range after the change. In this way, the control unit 21 can generate an AR image of the rainfall distribution image or the temperature distribution image corresponding to the photographed range of the earth ball 30 after the change, and display it superimposed on the photographed image of the earth ball 30. Therefore, for the rainfall distribution image or the temperature distribution image, an AR image corresponding to a place (location information) included in the photographed range is displayed superimposed on the photographed image of the earth ball 30 according to the photographed range of the earth ball 30 that is appropriately changed. In this case, too, when the shooting range of the Earth ball 30 is changed by the user holding the Earth ball 30 in his / her hand and rotating it, or by changing the shooting direction of the Earth ball 30 by the camera 26, the AR image (weather content) superimposed on the captured image of the Earth ball 30 is updated in real time according to the changed shooting range.

[0048] When the control unit 21 determines that the shooting range has not been changed (S41: NO), it determines whether or not an instruction to end the process has been given, for example, via the input unit 24 (S42). When the control unit 21 determines that an instruction to end the process has not been given (S42: NO), it returns to step S32 and repeats the above-mentioned process. When the control unit 21 determines that an instruction to end the process has been given (S42: YES), it ends the process. Through the above-mentioned process, weather content related to the weather on the earth, which changes from moment to moment, can be confirmed in the AR image displayed superimposed on the captured image of the earth ball 30. In addition, since the user can specify the location where the user wants to view the weather content by photographing the location on the earth ball 30, the weather content desired by the user can be easily displayed. In addition, since the user can change the shooting range of the earth ball 30 by holding and rotating the earth ball 30 during shooting, the weather content of the location desired by the user can be updated in real time even while the AR image is being displayed.

[0049] Next, a process of displaying an AR image using a border line image for displaying a border line and a country name on the captured image of the Earth ball 30 during the above-mentioned display process of the AR image will be described. The user terminal 20 of this embodiment is configured to be able to switch whether to display the AR image of the border line image on the captured image of the Earth ball 30 and the AR image of the weather content by superimposing it on the AR image or not, depending on the user's operation via the input unit 24. FIG. 11 is a flowchart showing an example of a display process procedure of an AR image using a border line image, and FIG. 12 is a schematic diagram showing an example of a display of an AR image. Although not shown, the border line image is a map image in which the border line and the country names of each country are written on a rectangular map expressed by equirectangular projection, as in the cloud distribution image shown in the upper part of FIG. 9. The border line image is registered in advance in the AR application 22AP. The border line image may include the names of continents and oceans on the map, the names of each city, and the like in addition to the border line and the country name. The following process is executed by the control unit 21 in accordance with the control program 22P and the AR application 22AP stored in the storage unit 22 of the user terminal 20. A part of the following process may be realized by a dedicated hardware circuit.

[0050] The control unit 21 of the user terminal 20 performs the following processing when, for example, in step S30 or S40 in FIG. 6 and FIG. 7, an AR image using meteorological content such as a cloud distribution image, a rainfall distribution image, or a temperature distribution image is displayed superimposed on the image captured by the earth ball 30 (S51). The left side of FIG. 12 shows a screen in which a cloud distribution image (AR image) is displayed superimposed on the image captured by the earth ball 30, similar to the screen on the right side of FIG. 8. As shown in FIG. 12 and other figures, the display screen of the AR image displays a border line switching button B2 for selecting whether to display an AR image of a border line image superimposed on the image captured by the earth ball 30 or not. In the screen on the left side of FIG. 12, the border line switching button B2 indicates a state in which the border line image is not displayed (OFF state), and when a touch operation is performed by the user in this state, it is switched to a state in which the border line image is displayed superimposed (ON state) as in the screen on the right side of FIG. 12. Furthermore, when the user performs a touch operation on the border line switching button B2 while the ON state is selected as shown in the screen on the right side of Fig. 12, the border line switching button B2 switches to the OFF state as shown in the screen on the left side of Fig. 12. Therefore, each time the user performs a touch operation on the border line switching button B2, it can receive a switching command that alternately switches between displaying and hiding the border line image.

[0051] Therefore, when an AR image of weather content is displayed superimposed on the captured image of the Earth ball 30, the control unit 21 judges whether the border line switching button B2 has been operated (S52). When the control unit 21 judges that the border line switching button B2 has not been operated (S52: NO), the control unit 21 waits while performing other processes including the processes of Figs. 6 and 7 until it judges that the border line switching button B2 has been operated. When the control unit 21 judges that the border line switching button B2 has been operated (S52: YES), the control unit 21 judges whether an instruction to switch to the ON state in which the border line image is displayed has been received (S53). Specifically, when the border line switching button B2 is operated while the OFF state is selected, the control unit 21 accepts an instruction to switch to the ON state, and when the border line switching button B2 is operated while the ON state is selected, the control unit 21 accepts an instruction to switch to the OFF state.

[0052] When it is determined that the instruction to switch to the ON state has been received (S53: YES), that is, when the border line switching button B2 is operated in a state in which the OFF state in which the border line image is not displayed is selected as shown in the left side of FIG. 12, the control unit 21 reads out the border line image (S54). Then, the control unit 21 generates an AR image of the border line image to be displayed superimposed on the captured image of the earth ball 30 based on the shooting range of the earth ball 30 specified in step S28 or S38 in FIG. 6 and FIG. 7 (S55). Here, the control unit 21 extracts an image of a hemispherical region corresponding to the shooting range of the earth ball 30 from the data of the border line image, and transforms the extracted border line image of the hemispherical region into a circular image to display it in a circular shape on the captured image of the earth ball 30. As a result, the control unit 21 generates a circular border line image by extracting a part of the border line image from the rectangular border line image expressed by the equirectangular projection.

[0053] Then, the control unit 21 displays the AR image of the generated border line image superimposed on the photographed image of the earth ball 30 being displayed on the display unit 25 (S56). Here, the control unit 21 displays the AR image of the border line image superimposed on the photographed image of the earth ball 30 on which the AR image of the weather content was superimposed and displayed in step S30 or S40 in FIG. 6 and FIG. 7. The AR image of the border line image may be superimposed on the AR image of the weather content, or may be superimposed on the photographed image of the earth ball 30 under the AR image of the weather content. This allows the AR image of the weather content processed into a circle and the AR image of the border line image to be superimposed on the photographed image of the earth ball 30, as shown on the right side of FIG. 12.

[0054] The control unit 21 judges whether the shooting range of the Earth ball 30 has been changed based on the captured images of the Earth ball 30 captured sequentially by the camera 26 (S57). If it is judged that the shooting range of the Earth ball 30 has been changed (S57: YES), the control unit 21 acquires a captured image of the Earth ball 30 by the camera 26 for the changed shooting range (S58), specifies the shooting range of the Earth ball 30 based on the captured image (S59), and generates an AR image of the weather content corresponding to the changed shooting range of the Earth ball 30 (S60), similar to steps S27 to S29 or S37 to S39 in Figs. 6 and 7. Then, the control unit 21 returns to the process of step S55, generates an AR image of the border line image corresponding to the changed shooting range of the Earth ball 30 (S55), and displays the generated AR image of the weather content and the AR image of the border line image superimposed on the captured image of the Earth ball 30 being displayed (S56).

[0055] When it is determined that the shooting range of the earth ball 30 has not been changed (S57: NO), the control unit 21 determines whether or not an instruction to end the process has been given, for example, via the input unit 24 (S61). When it is determined that an instruction to end the process has not been given (S61: NO), the control unit 21 returns to step S57 and repeats the above-mentioned process. When it is determined that an instruction to end the process has been given (S61: YES), the control unit 21 ends the process. Through the above-mentioned process, each time the shooting range of the earth ball 30 photographed by the camera 26 of the user terminal 20 is changed, an AR image according to the shooting range of the earth ball 30 can be superimposed and displayed. In addition to the AR image of the weather content, an AR image of a border line image can be superimposed and displayed on the photographed image of the earth ball 30.

[0056] On the other hand, when it is determined that an instruction to switch to the OFF state has been received (S53: NO), that is, when the border line switching button B2 is operated while the ON state in which the border line image is displayed is selected as shown on the right side of Fig. 12, the control unit 21 ends the display of the AR image of the border line image (S62). That is, the control unit 21 displays only the AR image of the weather content superimposed on the captured image of the Earth ball 30.

[0057] Then, the control unit 21 judges whether the shooting range of the Earth ball 30 has been changed based on the captured images of the Earth ball 30 captured sequentially by the camera 26 (S63). If it is judged that the shooting range of the Earth ball 30 has been changed (S63: YES), the control unit 21 acquires a captured image of the Earth ball 30 by the camera 26 for the changed shooting range (S64), identifies the shooting range of the Earth ball 30 based on the captured image (S65), and generates an AR image of the weather content corresponding to the changed shooting range of the Earth ball 30 (S66), as in steps S58 to S60. Then, the control unit 21 displays the generated AR image of the weather content by superimposing it on the captured image of the Earth ball 30 being displayed (S67), and returns to the processing of step S63.

[0058] When it is determined that the shooting range of the Earth ball 30 has not been changed (S63: NO), the control unit 21 determines whether or not an instruction to end the process has been given via, for example, the input unit 24 (S68). When it is determined that an instruction to end the process has not been given (S68: NO), the control unit 21 returns to step S63 and repeats the above-mentioned process. When it is determined that an instruction to end the process has been given (S68: YES), the control unit 21 ends the process. When the AR image of the weather content is displayed superimposed on the shooting image of the Earth ball 30 by the above-mentioned process, it is possible to switch whether or not to additionally display the AR image of the border line image according to the user's operation via the input unit 24. Therefore, when the AR image of the border line image is also displayed, the weather content on the earth can be confirmed together with the border line and the country name, etc., and it is possible to provide weather content that is easy to view. Note that even when the AR image of the border line image is displayed, the AR image of the weather content according to the shooting range of the Earth ball 30 and the AR image of the border line image are switched and displayed in real time every time the shooting range of the Earth ball 30 is changed. Therefore, the user can easily specify the location where he or she wants to view the weather content by changing the shooting range of the earth ball 30.

[0059] Next, a process of switching between 24 hours' worth of weather content and one week's worth of weather content as the weather content to be displayed overlaid on the captured image of the Earth ball 30 during the execution of the above-mentioned AR image display process will be described. The user terminal 20 of this embodiment is configured to be able to switch the weather content to be displayed overlaid on the captured image of the Earth ball 30 between 24 hours' worth of weather content and one week's worth of weather content in response to a user's operation via the input unit 24. In the following, 24 hours' worth of weather content and one week's worth of weather content are used as the weather content to be displayed overlaid on the captured image of the Earth ball 30, but weather content for another predetermined period may also be used.

[0060] Fig. 13 is a flowchart showing an example of an AR image switching process, and Fig. 14 is a schematic diagram showing an example of an AR image display. In Fig. 13, the left side shows the process performed by the user terminal 20, and the right side shows the process performed by the server 10. The following process is executed by the control unit 21 in accordance with a control program 22P and an AR application 22AP stored in the storage unit 22 of the user terminal 20, and is executed by the control unit 11 in accordance with a control program 12P stored in the storage unit 12 of the server 10. Part of the following process may be realized by a dedicated hardware circuit.

[0061] The control unit 21 of the user terminal 20 performs the following processing when, for example, in step S30 or S40 in FIG. 6 and FIG. 7, an AR image using weather content such as a cloud distribution image, a rainfall distribution image, or a temperature distribution image is displayed superimposed on the captured image of the earth ball 30 (S71). The left side of FIG. 14 shows a screen in which an AR image of a cloud distribution image and an AR image of a border line image are displayed superimposed on the captured image of the earth ball 30, similar to the screen on the right side of FIG. 12. As shown in FIG. 14 and other figures, the display screen of the AR image displays a playback switch button B3 for selecting whether to display weather content (weather images) for 24 hours or weather content (weather images) for one week as the AR image superimposed on the captured image of the earth ball 30. In the screen on the left side of FIG. 14, the playback switch button B3 indicates that weather images for 24 hours are displayed, and when a touch operation is performed by the user in this state, it is switched to a playback operation button B4 for instructing playback processing of weather images for one week, as in the screen on the right side of FIG. 14. The playback operation button B4 is configured to be switched to a playback switching button B3 for instructing playback processing of weather images for 24 hours as shown in the screen on the left side of FIG. 14 when a predetermined operation is performed.

[0062] Therefore, when the control unit 21 displays an AR image of weather images (weather content) for 24 hours superimposed on the captured image of the Earth ball 30, the control unit 21 judges whether the playback switching button B3 has been operated (S72). When the control unit 21 judges that the playback switching button B3 has not been operated (S72: NO), the control unit 21 waits while performing other processes including the processes of Figs. 6 and 7 until it judges that the playback switching button B3 has been operated (S72: YES), the control unit 21 requests one week's worth of weather images for the weather information being displayed from the server 10 (S73). For example, when the AR image being displayed is a cloud distribution image (first weather image), the control unit 21 requests only one week's worth of cloud distribution images (first weather image) from the server 10. In addition, if the AR image being displayed is a rainfall distribution image (second weather image) or a temperature distribution image (third weather image), the control unit 21 requests only one week's worth of rainfall distribution images (second weather image) or temperature distribution images (third weather image) from the server 10.

[0063] When a week's worth of weather images is requested from the user terminal 20, the control unit 11 of the server 10 refers to the date and time stored in the weather content DB 12a and reads out the data of weather images for the past week from the current date and time from the weather content DB 12a (S74). For example, the control unit 11 extracts the date and time included in the past week from the current date and time from the date and time stored in the weather content DB, and reads out the data of weather images corresponding to the extracted date and time together with the date and time of each data. Note that the control unit 11 may read out data of one weather image every 12 hours or one weather image every 6 hours as data of weather images for one week, but is not limited thereto.

[0064] The control unit 11 transmits the data of weather images for one week read from the weather content DB 12a to the user terminal 20 via the network N (S75). At this time, the control unit 11 gives a file name including the date and time of each piece of weather image data read from the weather content DB 12a, and generates time-series data of weather images (moving images) by combining the data of weather images for one week in the order of the file names. Thus, the user terminal 20 can obtain (download) the time-series data of weather images expressed in equirectangular projection from the server 10. The control unit 21 of the user terminal 20 stores the time-series data of weather images (data of weather images for one week) obtained from the server 10 in the storage unit 22 (S76).

[0065] Then, the control unit 21 performs the same process as steps S27 to S29 or S37 to S39 in FIG. 6 and FIG. 7, acquires a photographed image of the Earth ball 30 by the camera 26 (S77), specifies the photographed range of the Earth ball 30 based on the photographed image (S78), and generates an AR image to be displayed superimposed on the photographed image of the Earth ball 30 from the data of weather images for one week acquired from the server 10 based on the specified photographed range (S79). Here, the control unit 21 generates a circular weather image based on each of the data of weather images for one week expressed in equirectangular projection, thereby generating time-series data (video data) of circular weather images for one week. Then, the control unit 21 generates a playback screen for playing the generated AR image (data of weather images for one week) and displays it on the display unit 25 (S80). As shown on the right side of FIG. 14, the playback screen displays the generated AR image superimposed on the photographed image of the Earth ball 30, and displays a playback operation button B4 for instructing the playback process of the AR image (weather images for one week). On the playback screen, the control unit 21 superimposes and displays one AR image (for example, the first AR image) of the generated AR images (time-series data of weather images for one week).

[0066] The play operation buttons B4 in the screen on the right side of FIG. 14 have, from the left, a play button for instructing to start playing an AR image, a frame back button for instructing to return to the previous AR image (the previous AR screen), and a frame forward button for instructing to display the next AR image (the next AR image), and above these buttons, a seek bar for specifying an arbitrary time (time) in one week's worth of weather images. The control unit 21 judges whether the play operation button B4 has been operated (S81), and when it judges that any button of the play operation button B4 has been operated (S81: YES), it performs a play process corresponding to the operated button (S82). For example, when the play button is operated, the control unit 21 sequentially displays one week's worth of weather images in order from the AR image being displayed superimposed on the captured image of the earth ball 30. The control unit 21 displays each weather image (AR image) in one week's worth of weather images in order from the past date and time to the future date and time. Furthermore, when the frame rewind button or frame forward button is operated, the control unit 21 changes the AR image displayed superimposed on the photographed image of the earth ball 30 to the weather image (AR image) immediately before or after the currently displayed AR image. Furthermore, when the seek bar is operated, the control unit 21 identifies the date and time specified by the seek bar within a week, and displays the weather image (AR image) corresponding to the identified date and time superimposed on the photographed image of the earth ball 30 on the display screen. This allows the control unit 21 to execute a playback process instructed via the playback operation button B4 for the AR image displayed superimposed on the photographed image of the earth ball 30.

[0067] When it is determined that the playback operation button B4 has not been operated (S81: NO), the control unit 21 skips step S82 and determines whether or not an instruction to end the process has been given via, for example, the input unit 24 (S83). When it is determined that an instruction to end the process has not been given (S83: NO), the control unit 21 returns to step S81 and repeats the playback process according to the operation performed on the playback operation button B4. When it is determined that an instruction to end the process has been given (S83: YES), the control unit 21 ends the process. Through the above-mentioned process, for each weather image of a cloud distribution image, a rainfall distribution image, or a temperature distribution image, it is possible to switch between displaying weather images for 24 hours or weather images for one week according to the user's operation via the input unit 24. Thus, it is possible to provide each weather image to the user depending on whether the user wants to see weather images for 24 hours or weather images for one week. As described above, even when 24-hour weather images or one-week weather images are displayed, each time the shooting range of the earth ball 30 is changed, the AR image of the weather content corresponding to the shooting range of the earth ball 30 is switched and displayed in real time. Therefore, when the user changes the shooting range of the earth ball 30 by holding and rotating the earth ball 30 in his / her hand or by changing the shooting direction of the earth ball 30 by the camera 26, the AR image (weather content) superimposed on the shot image of the earth ball 30 is updated in real time according to the changed shooting range.

[0068] In this embodiment, a weather image related to past weather is used for the AR image, but is not limited thereto. For example, a future cloud distribution image, a rainfall distribution image, or a temperature distribution image based on a weather forecast may be used to generate an AR image related to future weather. In addition, the content used for the AR image is not limited to a weather image, and content showing the state of the earth that changes from moment to moment may be used. For example, an AR image may be generated using a satellite photo of the earth taken from a satellite. In this case, the server 10 periodically downloads satellite photos from a web server that publishes satellite photos, and provides the satellite photos to the user terminal 20 in response to a request from the user terminal 20. As a result, the user terminal 20 can generate an AR image to be displayed by superimposing it on the captured image of the earth ball 30 using the satellite photo acquired from the server 10, and it becomes possible to display an AR image using a satellite photo.

[0069] (Embodiment 2) An AR system for photographing different types of earth balls as subjects will be described. Different types of earth balls are earth balls with different AR markers printed on the surface of the sphere. The different types of earth balls may differ in size, etc. The user terminal 20 of this embodiment recognizes markers for different types of earth balls using different marker recognition systems. Therefore, the AR application 22AP of this embodiment realizes a marker recognition process that recognizes each AR marker for each type of earth ball. The AR system of this embodiment can be realized by a device similar to the AR system of embodiment 1, so a detailed description of the configuration will be omitted.

[0070] FIG. 15 is a schematic diagram showing another example of the configuration of the Earth Ball. The Earth Ball 50 (globe) shown in FIG. 15 is, for example, a ball made of PVC (polyvinyl chloride) that becomes a sphere by filling it with air, but is not limited to this. A map of the Earth is printed on the surface of the Earth Ball 50, and AR markers are attached at appropriate positions on the map. In the example shown in FIG. 15, each of the seas and continents in the map is given a name, and each country and city is given a country name and city name. These letters are used as AR markers. Although omitted in FIG. 15, the map includes the borders of each country, and the borders are also used as AR markers. In addition, rivers, plains, mountain ranges, etc. in the map are colored with a predetermined color for each landform, and each color is also used as an AR marker. Furthermore, the seas in the map are given predetermined marks M1 to M4 (for example, ship marks M1 and M2, and animal marks M3 and M4 such as turtles or seagulls) to identify each sea, sea area, etc. Also, predetermined marks M5 to M7 are added to the continents in the map in order to identify each continent, region, etc. These marks M1 to M7 are also used as AR markers.

[0071] Regarding the earth ball 50, each AR marker printed on the surface of the earth ball 50 is associated with location information (longitude and latitude information) on the earth, and the correspondence information between each AR marker and the location information is set in the AR application 22AP. Therefore, when the earth ball 50 is photographed, the control unit 21 of the user terminal 20 can specify the photographing range of the earth ball 50 by specifying the location information (longitude and latitude information) associated with the AR marker recognized in the photographed image.

[0072] Fig. 16 is a flowchart showing an example of a display processing procedure for an AR image according to embodiment 2. The processing shown in Fig. 16 is obtained by adding step S91 between steps S22 and S23 and adding steps S92 to S94 instead of step S28 in the processing shown in Fig. 6 and Fig. 7. Explanations of the same steps as in Fig. 6 and Fig. 7 are omitted, and steps S31 to S42 in Fig. 6 and Fig. 7 are omitted in Fig. 16.

[0073] In the AR system of this embodiment, the control unit 21 of the user terminal 20 performs the process of steps S21 to S22 in FIG. 6. The control unit 21 then identifies the type of the earth ball to be photographed based on the photographed images of the earth ball sequentially acquired by photographing with the camera 26 (S91). Here, the control unit 21 identifies whether the earth ball to be photographed is an earth ball 30 on which no border line or country name is printed, or an earth ball 50 on which a border line, country name, and various marks M1 to M7 are printed. Specifically, the control unit 21 determines the presence or absence of a border line, a character, and a predetermined mark M1 to M7 on the earth ball in the photographed image, and identifies it as an earth ball 50 if it determines that any of them are present, and identifies it as an earth ball 30 if it determines that none of them are present.

[0074] Thereafter, the control unit 21 of the user terminal 20 and the control unit 11 of the server 10 perform the processes of steps S23 to S27 in Fig. 6. As a result, the user terminal 20 downloads 24 hours' worth of cloud distribution image data from the server 10 and stores it in the storage unit 22 (S26), and acquires an image of the Earth ball photographed by the camera 26 (S27). Next, the control unit 21 judges whether or not the Earth ball identified in step S91 was an unmarked Earth ball 30 that does not have various marks (such as marks M1 to M7 in Fig. 15) including borders and country names printed thereon (S92). That is, the control unit 21 judges whether or not the identified Earth ball was the Earth ball 30 shown in Fig. 1.

[0075] If it is determined that the Earth ball 30 does not have a mark (S92: YES), the control unit 21 performs the same process as step S28 in Fig. 6. Specifically, the control unit 21 recognizes the map (continent map 31 and ocean map 32) and cloud mark 33 in the photographed image of the Earth ball 30 acquired in step S27 as AR markers, and specifies the shooting range of the Earth ball 30 based on the recognized AR marker (S93). On the other hand, if it is determined that the Earth ball 50 has a mark printed thereon, including borders and country names (S92: NO), the control unit 21 proceeds to the process of step S94. Then, the control unit 21 recognizes the map in the photographed image of the Earth ball 50 acquired in step S27, the names of the sea, continent, country, city, etc. in the map, borders, colors of each landform, and various marks M1 to M7 as AR markers, and specifies the shooting range of the Earth ball 50 based on the recognized AR marker (S94).

[0076] Thereafter, the control unit 21 generates an AR image to be displayed superimposed on the captured image of the earth ball 30, 50 based on the data of the cloud distribution image acquired from the server 10 and the shooting range of the earth ball 30, 50 specified in step S93 or S94 (S29). Then, the control unit 21 performs the process after step S30. As a result, in this embodiment as well, an AR image of a meteorological image such as a cloud distribution image, a rainfall distribution image, or a temperature distribution image processed in a circular shape can be displayed superimposed on the captured image of the earth ball 30, 50. Note that the earth balls 30, 50 only differ in the AR marker used when specifying the respective shooting ranges, and the control unit 21, after specifying the shooting range of the earth ball 30, 50, can generate an AR image by the same process and display it superimposed on the captured image of the earth ball 30, 50. Also in this embodiment, when the shooting range of the Earth ball 30, 50 is changed by the user holding the Earth ball 30, 50 in his / her hand and rotating it, or by changing the shooting direction of the Earth ball 30, 50 by the camera 26, the AR image (weather content) displayed superimposed on the captured image of the Earth ball 30, 50 can be updated in real time according to the changed shooting range.

[0077] In this embodiment, the same effect as that of the first embodiment can be obtained. In addition, in this embodiment, the AR system can be used for different types of earth balls 30, 50 on which different AR markers are printed. Therefore, regardless of the type of earth ball 30, 50, by photographing the earth ball 30, 50, weather content related to the weather on the earth can be superimposed and displayed on the photographed image of the earth ball 30, 50. In addition, the user can specify the place on the earth where he / she wants to see the weather content by photographing any place on the earth ball 30, 50, so that the operability is good. In this embodiment, the modified example described as appropriate in the first embodiment can also be applied.

[0078] In each of the above-described embodiments, when the user terminal 20 starts the AR application 22AP, the weather content of the cloud distribution image is downloaded from the server 10 without downloading the weather content of the rainfall distribution image and the temperature distribution image from the server 10, and the cloud distribution image is used as an AR image, but the configuration is not limited to this. For example, when the user terminal 20 starts the AR application 22AP, the weather content of the rainfall distribution image or the temperature distribution image may be downloaded from the server 10, and the rainfall distribution image or the temperature distribution image may be used as an AR image.

[0079] The following supplementary notes are further disclosed regarding each of the above embodiments.

[0080] (Appendix 1) Download time series data of first-stage meteorological images showing first-stage meteorological information from around the world in equirectangular projection. Download time series data of secondary weather images showing secondary weather information of the world in equirectangular projection, Download time series data of tertiary weather images showing tertiary weather information of the world in equirectangular projection. Selectably displaying a first image icon, a second image icon, or a third image icon for selecting one of the first weather image, the second weather image, or the third weather image; Obtain a photograph of the globe from the camera, When the first image icon is selected, each first weather image in equirectangular projection is corrected so that it covers the entire globe in the captured image and matches the image drawn on the globe, and the corrected first weather images that are consecutive in time series are continuously played back. A program that causes a computer to carry out processing.

[0081] (Appendix 2) Download time series data of equirectangular cloud distribution images showing cloud distribution around the world. Download time series data of rainfall distribution images showing the distribution of rainfall around the world in equirectangular projection. Download time series data of temperature distribution images showing global temperature distribution in equirectangular projection. Selectably displaying a cloud image icon, a rainfall image icon, or a temperature image icon for selecting one of a cloud distribution image, a rainfall distribution image, or a temperature distribution image; Obtaining from a camera a photographed image of a globe on which no borders or country names are written, a continental map and an oceanic map, and a dummy cloud image is drawn on the oceanic map, or a globe on which borders and country names are written, and a continental map and an oceanic map are drawn; Based on the acquired photographed image, it is determined whether the photographed globe is a globe without borders and country names or a globe with borders and country names; If the globe does not have the borders and country names, a photographed area of ​​the globe in the photographed image is identified based on the continent map or a dummy cloud image, and if the globe has the borders and country names, a photographed area of ​​the globe in the photographed image is identified based on the continent map, the borders, or the country names. Based on the identified photographed area of ​​the globe, any one of the cloud distribution images, the rainfall distribution images, or the temperature distribution images of the equirectangular projection is corrected so as to cover the entire globe in the photographed image and match the image drawn on the globe, and the corrected cloud distribution images, the rainfall distribution images, or the temperature distribution images that are successive in time series are continuously reproduced. A program that causes a computer to carry out processing.

[0082] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the scope and meaning equivalent to the claims. [Explanation of symbols]

[0083] 10 Server 11 Control section 12 Storage section 13. Communications Department 20 User terminal 21 Control section 22 Memory section 23 Communications Department 25 Display section 26 Camera 30 Earth Ball 31 Continent Map 32 Oceanographic Charts 33 Cloud mark 40 Weather Server 50 Earth Ball 12a Weather Content DB

Claims

1. When a start command is received, a download request is made to a server computer for downloading time series data of a cloud distribution image showing a global cloud distribution in equirectangular projection over a predetermined period of time, and the time series data of the cloud distribution image is downloaded; Obtain a photograph of the globe from the camera, Correcting each downloaded equirectangular cloud distribution image so that it covers the entire globe in the captured image and matches the image drawn on the globe, and continuously playing back the corrected cloud distribution images that are successive in time series; Selectably displaying a cloud image icon, a rainfall image icon, or a temperature image icon for selecting one of a cloud distribution image, a rainfall distribution image, or a temperature distribution image; When the rainfall image icon or the temperature image icon is selected while the cloud distribution image is being continuously played back, a download request is made to a server computer for time series data over a predetermined period of a rainfall distribution image showing a global rainfall distribution in equirectangular projection, or time series data over a predetermined period of a temperature distribution image showing a global temperature distribution in equirectangular projection, and the time series data of the rainfall distribution image or the time series data of the temperature distribution image is downloaded; Each downloaded rainfall distribution image or each downloaded temperature distribution image is corrected so as to cover the entire globe in the captured image and match the image drawn on the globe, and the corrected rainfall distribution images or temperature distribution images that are successive in time series are continuously reproduced. A program that causes a computer to carry out processing.

2. Download time-series data of cloud distribution images showing the cloud distribution of the world in equirectangular projection, Download time series data of rainfall distribution images showing the distribution of rainfall around the world in equirectangular projection. Download time series data of temperature distribution images showing global temperature distribution in equirectangular projection. Selectably displaying a cloud image icon, a rainfall image icon, or a temperature image icon for selecting one of a cloud distribution image, a rainfall distribution image, or a temperature distribution image; Obtaining from a camera a photographed image of a globe on which no borders or country names are written, a continental map and an oceanic map, and a dummy cloud image is drawn on the oceanic map, or a globe on which borders and country names are written, and a continental map and an oceanic map are drawn; Based on the acquired photographed image, it is determined whether the photographed globe is a globe without borders and country names or a globe with borders and country names; If the determination result is a globe without the borders and country names, a photographed area of ​​the globe in the photographed image is identified based on the continent map or a dummy cloud image, and if the determination result is a globe with the borders and country names, a photographed area of ​​the globe in the photographed image is identified based on the continent map, the borders, or the country names; Based on the identified photographed area of ​​the globe, any one of the cloud distribution images, the rainfall distribution images, or the temperature distribution images of the equirectangular projection is corrected so as to cover the entire globe in the photographed image and match the image drawn on the globe, and the corrected cloud distribution images, the rainfall distribution images, or the temperature distribution images that are successive in time series are continuously reproduced. A program that causes a computer to carry out processing.

3. The globe is a globe without national borders or country names, receiving a switching command as to whether or not to superimpose national borders and country names on a globe in the captured image while the cloud distribution image, the rainfall distribution image, or the temperature distribution image is being continuously reproduced; When a switching command to superimpose and display national borders and country names is received, the national borders and country names are superimposed on the globe in the captured image, and the corrected cloud distribution image, precipitation distribution image, or temperature distribution image is displayed.

3. The program according to claim 1, which causes the computer to execute a process.

4. The globe is a globe having a continental map and an oceanographic map, and a dummy cloud image drawn on the oceanographic map; Identifying a photographing range of the globe in the acquired photographed image based on position information associated with the continent map or dummy cloud image in the acquired photographed image; Based on the identified shooting range, the cloud distribution image, the rainfall distribution image, or the temperature distribution image in the equirectangular projection is corrected so as to match an image drawn on a globe in the captured image.

4. The program according to claim 1, which causes the computer to execute a process.

5. When the image of the globe in the captured image is changed while the cloud distribution image, the rainfall distribution image, or the temperature distribution image is being continuously played back, the captured range of the globe in the captured image after the change is identified in real time; The cloud distribution image, the rainfall distribution image, or the temperature distribution image of the equirectangular projection is corrected based on the identified shooting range.

5. The program according to claim 1, which causes the computer to execute a process.

6. An information processing device including a processor and a display unit, The processor, When the start command is received, a download request is made to the server computer for downloading time series data of a cloud distribution image showing a global cloud distribution in equirectangular projection over a predetermined period of time, and the time series data of the cloud distribution image is downloaded; Obtain a photograph of the globe from the camera, Correcting each downloaded equirectangular cloud distribution image so that it covers the entire globe in the captured image and matches the image drawn on the globe, and continuously playing back the corrected cloud distribution images that are successive in time series; displaying, on the display unit, a cloud image icon, a rainfall image icon, or a temperature image icon in a selectable manner for selecting one of a cloud distribution image, a rainfall distribution image, or a temperature distribution image; When the rainfall image icon or the temperature image icon is selected while the cloud distribution image is being continuously played back, a download request is made to a server computer for time series data over a predetermined period of a rainfall distribution image showing a global rainfall distribution in equirectangular projection, or time series data over a predetermined period of a temperature distribution image showing a global temperature distribution in equirectangular projection, and the time series data of the rainfall distribution image or the time series data of the temperature distribution image is downloaded; Each downloaded rainfall distribution image or each downloaded temperature distribution image is corrected so as to cover the entire globe in the captured image and match the image drawn on the globe, and the corrected rainfall distribution images or temperature distribution images that are successive in time series are continuously reproduced. Information processing device.

7. When a start command is received, a download request is made to a server computer for downloading time series data of a cloud distribution image showing a global cloud distribution in equirectangular projection over a predetermined period of time, and the time series data of the cloud distribution image is downloaded; Obtain a photograph of the globe from the camera, Correcting each downloaded equirectangular cloud distribution image so that it covers the entire globe in the captured image and matches the image drawn on the globe, and continuously playing back the corrected cloud distribution images that are successive in time series; Selectably displaying a cloud image icon, a rainfall image icon, or a temperature image icon for selecting one of a cloud distribution image, a rainfall distribution image, or a temperature distribution image; When the rainfall image icon or the temperature image icon is selected while the cloud distribution image is being continuously played back, a download request is made to a server computer for time series data over a predetermined period of a rainfall distribution image showing a global rainfall distribution in equirectangular projection, or time series data over a predetermined period of a temperature distribution image showing a global temperature distribution in equirectangular projection, and the time series data of the rainfall distribution image or the time series data of the temperature distribution image is downloaded; Each downloaded rainfall distribution image or each downloaded temperature distribution image is corrected so as to cover the entire globe in the captured image and match the image drawn on the globe, and the corrected rainfall distribution images or temperature distribution images that are successive in time series are continuously reproduced. An information processing method in which processing is performed by a computer.

8. The user is provided with a globe without borders or country names. Downloading time series data of cloud distribution images showing cloud distributions around the world in equirectangular projection from a weather server that publishes meteorological information, adding date and time information to the cloud distribution images and storing them in a storage unit; downloading time series data of a rainfall distribution image showing a global rainfall distribution in equirectangular projection from the weather server, adding date and time information to the rainfall distribution image and storing it in a storage unit; downloading time series data of a temperature distribution image showing the temperature distribution of the world in equirectangular projection from the weather server, adding date and time information to the temperature distribution image and storing it in a storage unit; When a start-up signal is received from a terminal device on which the user has installed an application, the time series data of the rainfall distribution image and the time series data of the temperature distribution image are not read out, but the time series data of the cloud distribution image for a predetermined period starting from the current date and time is read out by referring to the date and time information stored in the storage unit; The read time series data of the cloud distribution image is transmitted to the terminal device. An information processing method in which processing is performed by a computer.

Images