Portable information terminal, information presentation system, and information presentation method

Abstract

The invention relates to a portable information terminal, an information presentation system and an information presentation method. When the portable information terminal stores the captured image data in the information server, the portable information terminal transmits related information at the time of imaging to the information server. In addition, when the image data stored in the information server is requested, the current terminal information and an image search condition in which the playback condition of the image data is described are transmitted to the information server. And the information server retrieves corresponding image data according to the current terminal information and the image retrieval condition and sends the image data to the portable information terminal. The portable information terminal displays the received image data as a related image on a display unit. At this time, the related information at the time of imaging is compared with the current terminal information, and the size of the related image is changed and displayed in accordance with the degree of matching of the information. Therefore, the user can effectively remember the state and impression at the time of shooting.

Description

[0001] This divisional application is a divisional application based on the Chinese patent application with application number 201880089242.9, application date February 16, 2018, and invention name “Portable information terminal, information prompt system and information prompt method”. Technical Field

[0002] The present invention relates to a portable information terminal, an information prompting system and an information prompting method for taking pictures and storing them in an information server and acquiring and displaying images from the information server. Background Art

[0003] The popularity of portable information terminals such as smartphones and tablet terminals has been increasing. Portable information terminals are equipped with camera functions, and users can use the camera functions to take various images and save the taken images to a storage device section of the portable information terminal, a cloud storage device, etc. In addition, by using the portable information terminal to play back and display the images saved in the storage device section, the cloud storage device, etc., the user can recall the conditions and impressions when the images were taken.

[0004] For example, Patent Document 1 takes "when a user actually comes to the place or stays at home, etc., he or she can experience the conditions and status of other people in the place, feel a relaxed relationship with the person, or relive his or her past conditions and status in the place and recall the past" as the subject, and discloses an information presentation method with the following structure, namely, "when the current position is measured or indicated by a user terminal and sent to a server, the server sends to the user terminal image information of a photo taken at a position within a certain distance relative to the current position, information indicating the shooting position, and biological information of the person taking the photo at the time of shooting. In the user terminal, the sent information is displayed by spatial expression, or presented by vibration or flashing of light."

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2009-187233 Summary of the invention

[0008] In the technology described in Patent Document 1, an image is selected based on the current position (or set position) and the shooting position and the distance therebetween according to the user's operation instruction, and the image is presented together with the biological information. At this time, the position information of the current position and the shooting position is used as the image selection condition, but it is generally believed that it is difficult to effectively provide an image that reminds the user of a memory only by this condition.

[0009] That is, in order to effectively remind the user of the memories of the time when the photos were taken, it is necessary to select images by taking into account various conditions such as the season (month and day) and time when the images were taken in the past, and the direction to which the photographed object (subject) was pointed. For example, if an image taken on a certain day in winter is displayed in winter (or on the same month and day), it is easy to recall the memories of that time. In addition, if an image taken in the evening of a certain day is displayed in the evening time period (evening), it is easy to recall the memories of that time. In addition, even after the photographed object (subject) photographed in the past disappears, if the image is displayed when facing the direction from the shooting position at that time to where the photographed object (subject) was, it is easy to recall the memories of that time. Moreover, such selection conditions (season, month and day, time, direction, etc.) vary according to the user and the situation in which the user is in, so it is best for the user to be able to set them appropriately.

[0010] In addition, when the selected memory image is presented to the user, it is necessary to design a display method thereof. In the aforementioned patent document 1, images captured in the past are linked to a map and displayed, and biological information acquired from the user when the images were captured is also presented. However, currently, when the user uses a portable terminal to capture images, it is not considered how to display the memory image for the camera image being captured. That is, when selecting multiple images, it is better not to display them uniformly, but to emphasize the images that are considered to be of particular interest to the user.

[0011] An object of the present invention is to provide a portable information terminal, an information presentation system, and an information presentation method for selecting images previously photographed and stored by a user according to conditions set by the user, and effectively displaying the images to the user.

[0012] The present invention provides a portable information terminal, comprising: an image input unit for inputting captured image data; an information acquisition unit for acquiring various information including date and time information, location information, direction information, and distance information to a subject as the status of the portable information terminal; a communication unit for sending and receiving image data and various information with an information server via a network; a display unit for displaying an input image based on the image data input by the image input unit and / or an associated image based on the image data received by the communication unit; and a control unit for controlling the actions of various units of the portable information terminal. The control unit controls so that when the captured image data is saved to the information server, it is sent together with the related information at the time of shooting, and the related information at the time of shooting includes: various information indicating the status of the portable information terminal at the time of shooting obtained by the information acquisition unit; and user identification information capable of identifying the user of the portable information terminal. When the image data saved in the information server is requested, the current terminal information and the image retrieval condition recording the playback condition of the image data set by the user of the portable information terminal are sent to the information server, and the current terminal information includes: various information indicating the current status of the portable information terminal obtained by the information acquisition unit; and user identification information capable of identifying the user of the portable information terminal. The image data retrieved by the information server based on the current terminal information and the image retrieval condition is acquired and displayed on the display unit as a related image.

[0013] In the information prompting system of the present invention, the portable information terminal comprises: an image input unit for inputting captured image data; an information acquisition unit for acquiring various information including date and time information, position information, direction information, and distance information to a subject as the status of the portable information terminal; a communication unit for transmitting and receiving image data and various information to and from an information server via a network; and a display unit for displaying an input image based on the image data input by the image input unit and / or a related image based on the image data received by the communication unit. The information server comprises: a storage unit for storing image data and various information sent from the portable information terminal; an image retrieval unit for retrieving corresponding image data from the storage unit according to a request from the portable information terminal; and a communication unit for transmitting and receiving image data and various information to and from the portable information terminal via a network.

[0014] When the portable information terminal saves the captured image data to the information server, it sends the captured image data together with the related information at the time of shooting, the related information at the time of shooting includes: various information indicating the status of the portable information terminal at the time of shooting acquired by the information acquisition unit; and user identification information capable of identifying the user of the portable information terminal. When requesting the image data saved in the information server, the current terminal information and the image retrieval condition describing the playback condition of the image data set by the user of the portable information terminal are sent to the information server, the current terminal information includes: various information indicating the current status of the portable information terminal acquired by the information acquisition unit; and user identification information capable of identifying the user of the portable information terminal. The information server retrieves the corresponding image data from the storage device unit according to the current terminal information and the image retrieval condition received from the portable information terminal, and sends the retrieved image data to the portable information terminal.

[0015] The present invention provides an information prompt method, comprising: a step in which a portable information terminal, when saving captured image data to an information server, sends the captured image data together with associated information at the time of shooting to an information server, wherein the associated information at the time of shooting includes: various information indicating the status of the portable information terminal at the time of shooting, such as date and time information, location information, direction information, and distance information to a subject; and user identification information that can identify a user of the portable information terminal; and a step in which the information server saves the image data sent from the portable information terminal and the associated information at the time of shooting to a storage device unit. In addition, the invention comprises the following steps: when the portable information terminal requests image data stored in the information server, the portable information terminal sends the current terminal information and the image retrieval condition recording the playback condition of the image data set by the user of the portable information terminal to the information server, wherein the current terminal information includes: various information indicating the current status of the portable information terminal; and user identification information capable of identifying the user of the portable information terminal; the information server retrieves the corresponding image data from the storage device unit based on the current terminal information and the image retrieval condition received from the portable information terminal, and sends the retrieved image data to the portable information terminal; and the portable information terminal displays the image data sent from the information server as an associated image on the display unit.

[0016] According to the present invention, images captured and saved by the user in the past are selected using conditions set by the user, and closely related images are highlighted and displayed, so the user can effectively recall memories such as the conditions and impressions at the time of shooting. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a structural diagram of the information prompt system.

[0018] Figure 2AThis is an appearance diagram of a portable information terminal.

[0019] Figure 2B It is a figure which shows the home screen of a portable information terminal.

[0020] Figure 2C It is a hardware structure diagram of a portable information terminal.

[0021] Figure 2D It is a software structure diagram of a portable information terminal.

[0022] Figure 3 It is the hardware structure diagram of the information server.

[0023] Figure 4A It is a diagram showing a memory control setting screen of the portable information terminal.

[0024] Figure 4B This is a diagram for explaining the details of registration condition setting.

[0025] Figure 4C This is a diagram for explaining the details of playback condition setting.

[0026] Figure 4D This is a diagram for explaining a method of calculating the direction of an object.

[0027] Figure 5A The figure shows a camera function screen of a portable information terminal.

[0028] Figure 5B This is a flowchart of the memory image registration process.

[0029] Fig. 6A It is a structural diagram of the image database in the information server.

[0030] Figure 6B It is a structural diagram of the user database in the information server.

[0031] Figure 6C It is a structural diagram of the related information database in the information server.

[0032] Fig. 7A This is a flowchart of the memory image playback process (on the portable information terminal side).

[0033] Figure 7B This is a flowchart of the memory image search process (information server side).

[0034] Fig. 8A It is a figure which shows an example of the display screen of a recall image (display example 1).

[0035] Figure 8B It is a diagram showing an example of a display screen of a recall image (display example 2).

[0036] Figure 8C It is a figure which shows an example of the display screen of a recall image (display example 3).

[0037] Fig. 9A This is a diagram showing a usage state of a viewing device inserted into a portable information terminal.

[0038] Fig. 9B This is a diagram for explaining changes in the user's line of sight and the pointing direction of the portable information terminal.

[0039] (Explanation of symbols)

[0040] 10: viewing device; 100: portable information terminal; 104: RAM; 110: storage device unit; 131: display unit; 133: first image input unit; 134: second image input unit; 150: sensor unit; 160: communication unit; 180: touch screen; 180b: recall image control setting screen; 180c: camera function screen; 180d~f: examples of display screens; 200: network; 300: information server; 304: RAM; 310: storage device unit; 360: LAN communication unit; 1102: shooting function unit; 1103: information acquisition unit; 1104: recall image control unit; 3101: information management unit; 3102: image retrieval unit; 3011: image data storage area; 3012: related information storage area. DETAILED DESCRIPTION

[0041] Hereinafter, examples of embodiments of the present invention will be described using the accompanying drawings. In the embodiments described below, the term "recall image" is used to refer to an image that allows the user who took the image to recreate the situation and impression at the time of shooting and recall the memory. Specifically, there are many cases where various conditions at the time of shooting (date, time, location, etc.) are associated with the current situation (date, time, location, etc.), which can generally be called "associated images."

[0042] The portable information terminal used in this embodiment may be a portable phone, a smart phone, a tablet terminal, etc. Alternatively, it may be a head-mounted information terminal such as a head-mounted display (HMD=Head Mounted Display). Furthermore, it may be a structure in which a smart phone, a tablet terminal, etc. are fixed to an adapter to replace the head-mounted information terminal. In addition, it may be a PDA (Personal Digital Assistants), a notebook PC (Personal Computer), an e-book reader, etc. In addition, it may be other digital devices such as a digital still camera, a video camera that can shoot moving images, a portable game console, etc. In the following embodiments, the case of using a smart phone as a portable information terminal is mainly described.

[0043] [Information prompt system structure]

[0044] Figure 1 1 is a system configuration diagram showing an example of the information presentation system of the present embodiment. The information presentation system is composed of a portable information terminal 100, a broadband network 200 such as the Internet, and an information server 300.

[0045] The portable information terminal 100 can be connected to the network (Internet) 200 via an access point or a base station of a mobile phone communication network not shown in the figure, and can send and receive data with each server device on the network 200. The connection between the portable information terminal 100 and the access point is carried out by a wireless communication method such as Wi-Fi (registered trademark) and / or other communication methods. The connection between the portable information terminal 100 and the base station of the mobile phone communication network is carried out by a W-CDMA (Wideband Code Division Multiple Access) (registered trademark) method, a GSM (Global System for Mobile communications) (registered trademark) method, a LTE (Long Term Evolution) method or other communication methods.

[0046] The information server 300 is a server device installed by a carrier that provides information storage / presentation services. The information server 300 receives and stores image data and related information sent from the portable information terminal 100 , and sends image data and related information retrieved in response to a request from the portable information terminal 100 to the portable information terminal 100 .

[0047] In addition, Figure 1In the figure, only one portable information terminal 100 and one information server 300 connected to the network 200 are described respectively, but a plurality of portable information terminals 100 and a plurality of information servers 300 can be connected as appropriate to transmit and receive data to and from each other.

[0048] [Appearance of portable information terminal]

[0049] Figure 2A 1 is an external view showing an example of the portable information terminal 100. Here, a front view (a) and a back view (b) are shown when the portable information terminal 100 is a smartphone.

[0050] In (a), the front surface of the portable information terminal 100 is provided with: an action indicator 124, a first image input unit 133, a monaural speaker 142, and a touch screen 180. The action indicator 124 displays the action state of the portable information terminal 100 by lighting up or blinking an LED (Light Emitting Diode). The touch screen 180 is composed of a touch panel 123 and a display unit 131.

[0051] In (b), the touch sensor 122, the second image input unit 134, the auxiliary light emission / infrared range finder 135, and the stereo speaker 143 are provided on the back of the portable information terminal 100. When an image is input from the second image input unit 134, the auxiliary light emission / infrared range finder 135 emits auxiliary light for compensating for insufficient light. In addition, the auxiliary light emission / infrared range finder 135 uses infrared rays to measure the distance to the subject.

[0052] Here, of the two image input units, the first image input unit 133 disposed on the front surface is referred to as an "internal camera", and the second image input unit 134 disposed on the back surface is referred to as an "external camera". The touch sensor 122 may be disposed not on the back surface of the portable information terminal 100, but on the side surface, the lower part of the front surface (the part that does not overlap with the touch screen 180), etc. In addition, the touch panel 123 constituting the touch screen 180 may also have the function of the touch sensor 122. In this case, the function of the touch sensor 122 (for example, the fingerprint authentication function) can be executed at any position on the touch screen 180.

[0053] The upper end of the portable information terminal 100 includes a power key 121 p as one of the operation keys 121 . The lower end of the portable information terminal 100 includes a voice input unit 145 and a μ-USB input unit 170 u as one of the expansion interface units 170 .

[0054] [Home screen of portable information terminal]

[0055] Figure 2B 1 is a diagram showing an example of a home screen displayed on the portable information terminal 100. The home screen 180a displayed on the touch screen 180 is composed of a main function icon display area 180a1, a general icon display area 180a2, an other information display area 180a3, a control key area 180a4, and a notification area 180a5. The home screen 180a is a basic screen that is displayed when the power of the portable information terminal 100 is turned on (ON) or the sleep state is released, when the home key is touched during the execution of any application, etc.

[0056] The main function icon display area 180a1 is a display area for icons associated with the main applications frequently used in the portable information terminal 100. The general icon display area 180a2 is a display area for icons associated with other applications. The other information display area 180a3 is an area for displaying general information such as time information and weather information. The control key area 180a4 is an area for displaying the "back key", "home key", and "APP history key". The notification area 180a5 is an area for notifying information such as the radio wave status and the remaining battery level.

[0057] [Hardware structure of portable information terminal]

[0058] Figure 2C 1 is a diagram showing an example of the hardware configuration of the portable information terminal 100. The portable information terminal 100 is configured to include a main control unit 101, a system bus 102, a ROM 103, a RAM 104, a storage device unit 110, an operation input unit 120, an image processing unit 130, a voice processing unit 140, a sensor unit 150, a communication unit 160, and an expansion interface unit 170.

[0059] The main control unit 101 is a microprocessor unit that controls the entire portable information terminal 100 according to a predetermined operation program. The system bus 102 is a data communication path for transmitting and receiving various commands and data between the main control unit 101 and each operation block in the portable information terminal 100.

[0060] ROM (Read Only Memory) 103 is a memory that stores basic operation programs such as operating systems and other operation programs (applications, the same below). For example, a rewritable ROM such as EEPROM (Electrically Erasable Programmable ROM) or flash ROM is used. RAM (Random Access Memory) 104 is a work area when the basic operation programs and other operation programs are executed. ROM 103 and RAM 104 can also be integrated with the main control unit 101. In addition, ROM 103 may not be Figure 2C Instead of the independent structure shown, a partial storage area in the storage device unit 110 is used.

[0061] The storage device unit 110 stores the action program of the portable information terminal 100, action setting values, personal information of the authorized user of the portable information terminal 100, authentication information, etc. In addition, it is possible to store the action program downloaded from the network, various data generated by the action program, etc. In addition, it is possible to store the content such as moving images, static images, and voices downloaded from the network. In addition, it is possible to store data such as moving images and static images captured using the camera function. It is also possible to replace all or part of the functions of the ROM 103 with a partial area of ​​the storage device unit 110. In addition, the storage device unit 110 needs to keep the stored information even when the portable information terminal 100 is not powered from the outside. Therefore, for example, a semiconductor element memory such as a flash ROM, an SSD (Solid State Drive), a disk drive such as an HDD (Hard Disc Drive), etc. is used. In addition, each action program stored in the ROM 103 and the storage device unit 110 can be updated and expanded in function through download processing from each server device on the network.

[0062] The operation input unit 120 is an instruction input unit for inputting operation instructions for the portable information terminal 100. The operation input unit 120 is composed of an operation key 121 arranged with a push button switch, a touch sensor 122 that detects the touch of the operator's (user's) finger according to the change of electrostatic capacitance, and a touch panel 123 overlapped with the display unit 131. Furthermore, as other operating devices, a keyboard connected to the expansion interface unit 170, an independent portable terminal device connected by wired communication or wireless communication, etc. can also be used. Alternatively, the portable information terminal 100 can also be operated by voice input. In addition, the touch sensor 122 has a function of detecting the fingerprint or palm print of the finger touching the sensor unit.

[0063] The image processing unit 130 is composed of a display unit 131, an image signal processing unit 132, a first image input unit 133, and a second image input unit 134. The display unit 131 is a display device such as a liquid crystal panel, which displays the image data processed by the image signal processing unit 132 and provides it to the user of the portable information terminal 100. The image signal processing unit 132 has a video RAM (not shown) and drives the display unit 131 according to the input image data. In addition, the image signal processing unit 132 performs decoding processing of the encoded image signal, format conversion processing, menu and other OSD (On Screen Display) signal superposition processing, etc. as needed. The first image input unit 133 and the second image input unit 134 are shooting components such as built-in cameras and external cameras, and use electronic devices such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor) sensors to convert light input from the lens into electrical signals and capture image data of the subject.

[0064] The voice processing unit 140 is composed of a voice output unit 141, a voice signal processing unit 144, and a voice input unit 145. The voice output unit 141 is a speaker, and provides the voice signal processed by the voice signal processing unit 144 to the user of the portable information terminal 100. The monaural speaker 142 performs monaural voice output during voice calls, and the stereo speaker 143 performs stereo voice output during music playback. The voice signal processing unit 144 performs decoding processing of encoded voice signals as needed. The voice input unit 145 is a microphone, and converts the user's voice into voice data for input.

[0065] The sensor unit 150 is a group of various sensors for detecting the state of the portable information terminal 100. As sensors, it has a GPS (Global Positioning System) receiving unit 151, a gyro sensor 152, a geomagnetic sensor 153, and an acceleration sensor 154 to detect the position, tilt, orientation, and movement of the portable information terminal 100. In addition, it can also have an illumination sensor, a proximity sensor, an air pressure sensor, etc. to detect the brightness of the surroundings, the proximity of surrounding objects, etc.

[0066] The communication unit 160 is composed of a LAN (Local Area Network) communication unit 161, a telephone network communication unit 162, and an NFC (Near Field Communication) unit 163. The LAN communication unit 161 is connected to a network 200 such as the Internet via an access point, etc., and transmits and receives data with an information server 300 on the network. The connection with the access point, etc. is performed using a wireless connection such as Wi-Fi (registered trademark). The telephone network communication unit 162 performs telephone communication (calls) and data transmission and reception through wireless communication with a base station of a mobile phone communication network, etc. The communication with the base station, etc. can be performed using a W-CDMA (registered trademark) method, an LTE method, etc. The NFC unit 163 performs wireless communication when it is close to a corresponding reader / writer. The LAN communication unit 161, the telephone network communication unit 162, and the NFC unit 163 are respectively equipped with an encoding circuit, a decoding circuit, an antenna, etc. The communication unit 160 may also be equipped with a Bluetooth (registered trademark) communication unit, an infrared communication unit, etc.

[0067] The expansion interface unit 170 is an interface group for expanding the functions of the portable information terminal 100, and is composed of an image / voice interface, a USB (Universal Serial Bus) interface, a memory interface, etc. The image / voice interface inputs / outputs image signals / voice signals with external image / voice devices. The USB interface is connected to a PC, a keyboard, and other USB devices to send and receive data. The memory interface is connected to a memory card and other memory media to send and receive data.

[0068] also, Figure 2C The illustrated configuration example of the portable information terminal 100 also includes a large number of configurations that are not essential to the operation of the present embodiment, and even if these configurations are not included, the effects of the present embodiment will not be affected.

[0069] [Software structure of portable information terminal]

[0070] Figure 2D 1 is a diagram showing an example of the software structure of the portable information terminal 100. Here, the structure of the software in the storage device unit 110 (or ROM103, the same below) and RAM104 is shown. The storage device unit 110 stores a basic operation program 1001, a shooting program 1002, an information acquisition program 1003, a recall image control program 1004, and other operation programs 1009. In addition, the storage device unit 110 has: an identification information storage area 1011 for storing user identification information that can identify the portable information terminal 100 or its user; and a various information storage area 1019 for storing other information.

[0071] The basic operation program 1001 stored in the storage device unit 110 is expanded in the RAM 104, and the expanded basic operation program is executed by the main control unit 101, thereby forming the basic operation unit 1101. Similarly, the shooting program 1002, the information acquisition program 1003, the recall image control program 1004, and the other operation programs 1009 are respectively expanded in the RAM 104, and the main control unit 101 executes the above-expanded operation programs, thereby forming the shooting function unit 1102, the information acquisition unit 1103, the recall image control unit 1104, and the other operation units 1109. In addition, the RAM 104 has a temporary storage area 1200 for temporarily storing data generated when each operation program is executed.

[0072] In addition, in order to simplify the description of the operation, the process of controlling each operation module by the main control unit 101 expanding the basic operation program 1001 stored in the storage device unit 110 in the RAM 104 and executing it is described as the process of controlling each operation module by the basic operation unit 1101. The operation of other functional modules performed by other operation programs is also described in the same way.

[0073] The image capturing function unit 1102 controls image capturing processing using camera components such as the first image input unit 133 and the second image input unit 134. In addition, since general operations and processing in image capturing processing are well known, detailed descriptions thereof will be omitted.

[0074] The information acquisition unit 1103 includes: a date and time information acquisition unit 1103A, a position information acquisition unit 1103B, an orientation information acquisition unit 1103C, and a distance information acquisition unit 1103D. The date and time information acquisition unit 1103A acquires date and time information from a built-in clock (not shown) and a standard time server connected to the network. The position information acquisition unit 1103B acquires the position information of the portable information terminal 100 based on the signal received by the GPS receiving unit 151. Alternatively, it is also possible to receive a beacon signal through a Bluetooth (registered trademark) communication unit. The orientation information acquisition unit 1103C acquires the orientation information of the portable information terminal 100 based on the geomagnetic information detected by the geomagnetic sensor 153. The distance information acquisition unit 1103D uses a known technique such as distance measurement performed using the auxiliary light emission / infrared rangefinder 135 to acquire the distance information between the portable information terminal 100 and the photographed object (subject).

[0075] The memory image control unit 1104 includes an image registration unit 1104A, an image request unit 1104B, and an image display unit 1104C. The image registration unit 1104A performs control for registering and storing an image taken by a user in the information server 300. The image request unit 1104B performs control for acquiring the registered and stored image from the information server 300 in accordance with the user's request. The image display unit 1104C performs control for displaying the image acquired from the information server 300 on the display unit 131.

[0076] In addition, each of the above-mentioned action programs is stored in the storage device unit 110 in advance at the time of product shipment. Alternatively, after the product is shipped, it can be obtained from each server device on the network via the LAN communication unit 161 or the telephone network communication unit 162. In addition, each of the above-mentioned action programs stored in a memory card, an optical disk, etc. can also be obtained via the expansion interface unit 170, etc.

[0077] [Hardware structure of information server]

[0078] Figure 3 1 is a diagram showing an example of the hardware configuration (including a part of the software configuration) of the information server 300. The information server 300 is composed of a main control unit 301, a system bus 302, a RAM 304, a storage device unit 310, and a LAN communication unit 360.

[0079] The main control unit 301 is a microprocessor component that controls the entire information server 300 according to a predetermined action program. The system bus 302 is a data communication path for sending and receiving various commands and data between the main control unit 301 and each action block in the information server 300. The RAM 304 is a work area when executing each action program. The RAM 304 can also be an integrated structure with the main control unit 301.

[0080] The storage device unit 310 stores an information management program 3001 and an image search program 3002. In addition, the storage device unit 310 includes an image data storage area 3011 and a related information storage area 3012. The information management program 3001 and the image search program 3002 stored in the storage device unit 310 are expanded in the RAM 304, and then the main control unit 301 executes the expanded information management program and the image search program 3002, thereby forming the information management unit 3101 and the image search unit 3102. In addition, the RAM 304 includes a temporary storage area (not shown) for temporarily storing data generated when executing each action program as needed. Here, the information management unit 3101 saves the memory image and its related information sent from the portable information terminal 100 in the image data storage area 3011 and the related information storage area 3012. The image search unit 3102 searches for the corresponding memory image in accordance with the request from the portable information terminal 100.

[0081] The LAN communication unit 360 is connected to a network 200 such as the Internet via a router device (not shown) and transmits and receives data to and from the portable information terminal 100 on the network. The connection to the Internet or the like via the router device may be a wired connection or a wireless connection. The LAN communication unit 360 includes an encoding circuit, a decoding circuit, and the like.

[0082] [Control Processing of Recall Images]

[0083] The following describes a control process for storing image data captured by the portable information terminal 100 as a memory image in the information server 300, and retrieving the memory image stored in the information server 300 and providing it to the portable information terminal 100 for playback and display. In addition, the "control process for the memory image" and the like may be referred to as the "memory control process" and the like.

[0084] [Recall control setting process]

[0085] Figure 4A 180b is a diagram showing an example of a control setting screen for recalling an image (recall control setting screen) in the portable information terminal 100. The recall control setting screen 180b is displayed by starting the recall control application. Figure 2B The user touches the icon of the memory control application displayed in the display area 180a1 or 180a2 of the main function icon or general icon on the home screen 180a to instruct the start of the memory control application. The basic operation unit 1101 transfers the control subject to the memory image control unit 1104 according to the user's instruction, and the memory image control unit 1104 displays the memory control setting screen 180b for setting the control conditions of the memory image on the touch screen 180 (display unit 131).

[0086] The memory control setting screen 180b is composed of a login setting area 180b1, a registration condition setting area 180b2, and a playback condition setting area 180b3. The login setting area 180b1 is an area for inputting login name information for logging into the information server 300 and password information for user authentication. After the memory control application is started, the portable information terminal 100 logs into the information server 300 using the information input into the login setting area 180b1.

[0087] The registration condition setting area 180b2 is an area for setting whether to register the image data captured using the camera function as a memory image, and whether to share the image data registered as a memory with other users. The playback condition setting area 180b3 is an area for setting what conditions are used to select and playback the memory images that were captured in the past and saved in the information server 300. As a specific selection condition, based on the information of the current terminal acquired by the portable information terminal 100 (current date and time information, current location information, current orientation information, etc.), the memory image of the shooting condition associated with it is selected. In addition, it is also possible to acquire and playback the image data registered as a memory by other users (friends). The user can simply check the desired condition item on the setting screen 180b. Furthermore, by touching the "Details" key displayed in the registration condition setting area 180b2 and the playback condition setting area 180b3, detailed conditions can be entered for each setting item.

[0088] The information input to the login setting area 180b1 is stored in the identification information storage area 1011 of the storage unit 110. The information input to the registration condition setting area 180b2 and the playback condition setting area 180b3 is stored in the various information storage area 1019 of the storage unit 110.

[0089] Figure 4B This is a diagram for explaining detailed settings in the registration condition setting area 180b2. Here, a case is described where a user checks the item "share image" to share image data registered as a memory image with other users.

[0090] In the "Shared Image" item, it is possible to select whether to "share the registered image data with registered users" or "share with all users", that is, it is possible to select the disclosure level of the image. In the former case of "share with registered users", the disclosure level is limited, and further, as the setting range, the users who are allowed to share the image data are registered (set). The registration of the user is specified using the e-mail address information, the login name information for logging in to the information server 300, etc. On the other hand, in the latter case of "share with all users", all users logged in to the information server 300 are the objects of sharing.

[0091] Figure 4C This is a diagram for explaining the detailed settings in the playback condition setting area 180b3. Here, the conditions under which the memory images stored in the information server 300 are played back and displayed are set in detail.

[0092] When the item "Replay by date and time" is checked, "year", "month", "day", and "time" are set separately as detailed conditions of date and time. In the case of replaying by "year", it is further possible to select, as a setting range, to replay only images shot in the same year as the current date and time, or to replay images shot more than or equal to X years ago from the current date and time. In the case of replaying by "month", it is further possible to select to replay only images shot in the same month as the current date and time, or to replay images shot within X months before and after the same month as the current date and time. In the case of replaying by "day", it is further possible to select to replay only images shot on the same day as the current date and time, or to replay images shot within X days before and after the same day as the current date and time. In the case of replaying by "time", it is further possible to set to replay the memory images shot within X minutes before and after the same time as the current time. In addition, the conditions of "year", "month", "day", and "time" regarding date and time may be used as a single condition or in combination with a plurality of conditions.

[0093] When the item "playback based on position" is checked, "current position" or "specified position" is set as a detailed condition. When playing back based on "current position", the X value can be set to play back the image captured at a position within Xm east-west and / or north-south from the current position of the portable information terminal 100. When playing back based on "specified position", the specified position is set using latitude / longitude, and the X value is set to play back the image captured at a position within Xm east-west and / or north-south from the specified position. Alternatively, the X value can be specified by the change range of latitude / longitude. It can also be specified to be within a radius of Xm from the specified position.

[0094] When the item "Replay by direction" is checked, "direction" or "when facing the direction of the object" is set separately as a detailed condition. In the case of replaying by "direction", the X value can be further set so that the image whose angle difference between the direction currently pointed by the portable information terminal 100 and the direction pointed at the time of past shooting is within ±X degrees is played back. In the case of replaying by "when facing the direction of the object", when the portable information terminal 100 faces the direction of the object (subject) in the image captured in the past, the image including the object is played back. Even in this case, the X value can be set so that the image whose angle difference is within ±X degrees is played back.

[0095] When the item "Replay friend's images" is checked, "specific user" or "all users" is set as a detailed condition. When replaying the images of "specific user", it is further set which user's images are to be replayed. The user is set using the e-mail address information or the login name information for logging into the information server 300. In addition, "specific user" can also be set with the Figure 4B When reproducing images of “all users”, images of all users registered in the information server 300 are the objects of reproduction.

[0096] In addition, each of the above-mentioned setting conditions for playback display (date and time, location, direction, friend's image) can be used alone or in combination. When multiple setting conditions are combined, logical expressions such as logical OR and logical AND can be used to more appropriately narrow the playback conditions. Thus, by combining the conditions of the shooting location and shooting direction instead of simply using the shooting date and time, it is possible to set playback conditions corresponding to the user's preferences, and it is possible to prompt a memory image that effectively reminds the user of the situation and impression at the time of shooting.

[0097] In addition, in “replay according to the orientation”, replaying the image including the object “when facing the direction of the object” is a particularly unique process in this embodiment, and its calculation method is described below.

[0098] Figure 4D This is a diagram for explaining a method of calculating the direction of an object observed from a portable information terminal.

[0099] Assume that the user has previously photographed the object S at the location A using the portable information terminal 100, and the photographed image is stored in the information server 300 as a memory image. At this time, the distance between the portable information terminal 100 and the object S is L, and the direction pointed by the portable information terminal 100 is α. In addition, the distance L is measured by the distance information acquisition unit 1103D of the information acquisition unit 1103. These photographing related information (position information of the location A, photographing direction α, distance L) are stored in the information server 300 together with the photographed image data of the object S.

[0100] On the other hand, the direction of the position of the object S when the current user is at the B location is described. The distance M and the angle β between AB can be calculated based on the position information obtained at the A location and the position information obtained at the B location. Based on this information (distance L, distance M, angle α, angle β), the direction γ of the object S observed from the B location and the distance N from the B location to the object S can be calculated through triangulation calculation. That is, when the current user is at the B location, if the direction pointed by the portable information terminal 100 is the calculated direction γ (or within the direction γ±X degrees), it can be determined that the portable information terminal 100 is facing the direction of the object S.

[0101] [Recall image registration process]

[0102] The following describes a memory image registration process in which an image of a subject is captured using the portable information terminal 100 and the captured image data is registered and stored in the information server 300 as a memory image together with its associated information.

[0103] Figure 5A 1 is a diagram showing an example of a camera function screen of the portable information terminal 100. Figure 2B The camera function screen 180c is displayed on the touch screen 180 by clicking the icon of the camera function application on the home screen 180a. The camera function screen 180c is composed of a camera control key area 180c1, a camera setting key area 180c3, and a focus position display mark 180c4. On the other hand, the image data input from the first image input unit (built-in camera) 133 or the second image input unit (external camera) 134 as the camera component is overlapped and displayed on the touch screen 180. The camera control key area 180c1 is provided with a shutter key 180c2. In addition, by touching the camera switching key prepared in the camera setting key area 180c3, the camera component is switched.

[0104] Figure 5B 2 is a flowchart showing the memory image registration process. This process is mainly controlled by the image registration unit 1104A of the memory image control unit 1104.

[0105] First, the user instructs to start the camera function application by operating the home screen 180a displayed on the portable information terminal 100. Specifically, the user touches the Figure 2B The basic operation unit 1101 confirms that the camera function application icon is displayed in the main function icon or general icon display area 180a1, 180a2. Figure 4AThe memory image registration setting 180b2 is set, and according to the user's instruction, the control subject is handed over to the image registration unit 1104A of the memory image control unit 1104. The image registration unit 1104A starts the shooting function unit 1102 (S101). The shooting function unit 1102 enables, for example, the second image input unit (external camera) 134 in the camera component, and starts to input image data from the second image input unit 134. Then, as shown in FIG. Figure 5A As shown, the photographing function section 1102 displays the image data input from the second image input section 134 and the camera function screen 180 c on the touch screen 180 (display section 131 ) ( S102 ).

[0106] Next, the basic operation unit 1101 checks whether there is an operation instruction input to the touch screen 180 (touch panel 123) based on the user's touch operation, and notifies the image registration unit 1104A of the confirmation result (S103). If there is no operation instruction input to the touch screen 180, it waits until the instruction is received. If an operation instruction is input to the touch screen 180, the type of the operation instruction is confirmed (S104).

[0107] If the type of the operation instruction confirmed in S104 is an instruction to end the camera function application, the process proceeds to S105, and the image registration unit 1104A ends the operation of the photographing function unit 1102, returning the control subject to the basic operation unit 1101. The basic operation unit 1101 displays the home screen 180a on the touch screen 180 (S106).

[0108] If the type of operation instruction confirmed in S104 is a shutter operation, the process proceeds to S107, and the shooting function unit 1102 starts the acquisition process of the image data input from the second image input unit 134. First, the shooting function unit 1102 acquires the image data from the second image input unit 134 and stores it in the temporary storage area 1200 of the RAM 104 (S107). The acquisition process of the image data includes: pre-processing such as focusing and exposure; processing of converting the output of electronic devices such as CCD and CMOS sensors into digital data; and post-processing such as gamma correction, noise removal, and image compression processing. Next, the image registration unit 1104A requests the information acquisition unit 1103 to acquire various information (date and time information, etc.) at the time of shooting.

[0109] Correspondingly, the date and time information acquisition unit 1103A acquires the shooting date and time information (year, month, day, hour, minute, second) when the image data is acquired (S108). The position information acquisition unit 1103B acquires the shooting position information (longitude, latitude) of the portable information terminal 100 when the image data is acquired (S109). The direction information acquisition unit 1103C acquires the shooting direction information to which the portable information terminal 100 is pointed when the image data is acquired (S110). Furthermore, the distance information acquisition unit 1103D acquires the subject distance information of the photographed object (subject) (S111).

[0110] Next, it is determined whether to register the captured image data as a memory image (S112). The user can Figure 4A Alternatively, the user may be able to make the setting from a menu or the like.

[0111] If it is determined in S112 that the image is to be registered as a memory image, the process proceeds to S113, where the image registration unit 1104A generates the associated information at the time of shooting associated with the registered image. The associated information at the time of shooting includes: user identification information that can identify the user of the portable information terminal 100 (or terminal identification information that can identify the portable information terminal 100); various information at the time of shooting (date and time information, position information, direction information, distance information) acquired by the information acquisition unit 1103; and the disclosure level setting information set in the "shared image" item of the registration condition setting area 180b2 of the memory control setting screen 180b. Furthermore, the image registration unit 1104A transmits the image data acquired in the process of S107 and the associated information at the time of shooting generated in the process of S113 to the information server 300 via the communication unit 160 and the network (Internet) 200 (S114).

[0112] In the information server 300, the transmitted image data and the associated information at the time of shooting are associated and stored in the image data storage area 3011 and the associated information storage area 3012. In addition, the image data acquired in the process of S107 and the associated information at the time of shooting generated in the process of S113 may also be stored together in the various information storage area 1019 of the storage device unit 110 in the portable information terminal 100. Then, the process returns to S103 and the above process is repeated.

[0113] On the other hand, in the case where the captured image data is not registered as a recall image in the determination of S112, the process proceeds to S115 to perform normal capturing processing. The image registration unit 1104A generates an image file (S115) based on the image data acquired in S107 and various information (date and time information, position information, direction information, distance information) acquired in S108 to S111 at the time of capturing. The image file may also be in the well-known EXIF ​​(Exchangeable Image File Format) format or the like. Furthermore, the image registration unit 1104A stores the image file generated in S115 in the various information storage area 1019 of the storage device unit 110 (S116). Then, the process returns to S103 and the above-mentioned process is repeated.

[0114] In addition, even if the captured image data is not registered as a memory image in the determination of S112, the image data acquired at this time and various information at the time of shooting are stored in the storage device unit 110. Therefore, the user can send the image as a memory image to the information server 300 for additional registration. In addition, the processing of S115 to S116 may be processed by the shooting function unit 1102 instead of the image registration unit 1104A as the control body.

[0115] [Database of information server]

[0116] The image data of the memory image and the related information at the time of shooting transmitted from the portable information terminal 100 are stored in the image data storage area 3011 and the related information storage area 3012 by the information management unit 3101 of the information server 300. Here, the structure in which these data are stored will be described.

[0117] Fig. 6A 1 is a diagram showing an example of an image database stored in the image data storage area 3011 of the information server 300. The structure of the image database is divided into folders for each user identification information that can identify the user of the portable information terminal 100, and the image data sent from the portable information terminal 100 is stored in the corresponding folder. Figure 6B )'s login name information and related information database ( Figure 6C ) to manage these folders. In addition, each image data in the folder is managed by the file name corresponding to the image identification information in the associated information database.

[0118] Figure 6B3012 is a diagram showing an example of a user database stored in the related information storage area 3012 of the information server 300. The user database is a database that stores a list of users who use the information storage / prompt service, and registers information such as login name, password, name, email address, and contact number. Other information such as address information and credit card information may also be registered.

[0119] The login name is identification information for identifying the user who uses the information storage / prompt service, and is used when logging in to the information storage / prompt service. The login name is uniquely determined for each user. The password is authentication information used for user authentication when logging in to the information storage / prompt service. Each user can arbitrarily specify a password. The name is information related to the name of the user who uses the information storage / prompt service. The email address and contact number are information related to the contact destination for each user who uses the information storage / prompt service, and are used when making various contacts. Figure 4A In the login setting area 180a1 shown, a login name and a password are inputted to perform login.

[0120] Figure 6C 30 is a diagram showing an example of a related information database stored in the related information storage area 3012 of the information server 300. The related information database is a database for storing related information at the time of shooting that is related to each image data transmitted from the portable information terminal 100.

[0121] The image identification information is identification information that can identify each image data sent from the portable information terminal 100. The user identification information is user identification information that can identify the person who photographed (registered) each image data, and is stored in the user database ( Figure 6B ) by combining the image identification information and the user identification information, the image data can be uniquely identified. The shooting date and time information is the shooting date and time information of each image data sent from the portable information terminal 100. Similarly, the shooting position information, shooting direction information, and subject distance information (not shown) of each image data are registered corresponding to the image identification information. Furthermore, information such as terminal identification information that can identify the portable information terminal and disclosure level setting information can also be registered.

[0122] [Replay image processing]

[0123] The following describes a memory image playback process for searching and acquiring memory images that were captured in the past and stored in the information server 300 based on the current status of the portable information terminal 100 and playing and displaying the memory images in the portable information terminal 100 .

[0124] Fig. 7A and Figure 7B is a flowchart showing the recall image playback process, Fig. 7AThe processing on the portable information terminal 100 side is shown. Figure 7B The processing on the information server 300 side is shown.

[0125] First, use Fig. 7A The image playback processing on the portable information terminal 100 side is described. The following processing is controlled by the image request unit 1104B of the recall image control unit 1104. In addition, it is possible to select whether to execute the recall image playback processing by setting the recall control application. Alternatively, it is also possible to set it from a user menu or the like. In the case of setting to execute the recall image playback processing, Fig. 7A The processing flow shown is always in operation.

[0126] In S201, the image request unit 1104B of the recall image control unit 1104 generates the user identification information of the portable information terminal 100 and the recall image search condition (hereinafter referred to as "image search condition") in which the playback condition set in the playback condition setting area 180b3 of the recall control setting screen 180b is described. Furthermore, the image request unit 1104B transmits the image search condition generated in the process of S201 to the information server 300 via the communication unit 160 and the network 200 (S202). The transmitted image search condition is stored in the related information storage area 3012 of the information server 300.

[0127] Next, the image request unit 1104B requests the information acquisition unit 1103 to acquire the current terminal status (date and time information, etc.). Correspondingly, the date and time information acquisition unit 1103A acquires the current date and time information (S203), and the position information acquisition unit 1103B acquires the current position information of the portable information terminal 100 (S204). Furthermore, the direction information acquisition unit 1103C acquires the direction information of the portable information terminal 100 currently pointing (S205).

[0128] Next, in S206, the image request unit 1104B generates information on the current terminal state (hereinafter referred to as "current terminal information") based on the current date and time information, current location information, current orientation information, and user identification information of the portable information terminal 100 acquired by the information acquisition unit 1103. Furthermore, the image request unit 1104B sends the current terminal information generated in S206 to the information server 300 via the communication unit 160 and the network 200 (S207).

[0129] Correspondingly, in the information server 300, as in Figure 7BAs described later, based on the image search condition (S202) sent from the portable information terminal 100 and the current terminal information (S207), the corresponding image data is retrieved from the database and sent to the portable information terminal 100 as a memory image together with the associated information at the time of shooting.

[0130] In S208, the image request unit 1104B of the portable information terminal 100 determines whether the memory image and its related information are received from the information server 300. When the memory image and related information are received, the operating status of the display unit 131 of the portable information terminal 100 is determined (S209), and the memory image display process (S210) or the memory image report process (S211) is executed. The processes of S210 and S211 are mainly controlled by the image display unit 1104C of the memory image control unit 1104.

[0131] In the determination of S209, when the display unit 131 is in operation (when the screen is being displayed), the image display unit 1104C determines that the user is in the process of using the portable information terminal 100. In this case, the process proceeds to S210, and the image display unit 1104C displays the received memory image on the display unit 131. The details of the display method of the memory image will be described later. Thereafter, the process proceeds to S212.

[0132] In the determination of S209, when the display unit 131 is not in operation (when no screen is displayed), the image display unit 1104C determines that the user is not using the portable information terminal 100. In this case, the process proceeds to S211, and the image display unit 1104C outputs a report sound for notifying that the memory image has been received from the stereo speaker 143, or vibrates the housing of the portable information terminal 100 through a vibrator (not shown). In this way, the user can know that the memory image has been received.

[0133] On the other hand, in the determination of S208, when the recollection image and related information are not received from the information server 300 (or when the information server 300 indicates that there is no image data to be replayed), the process proceeds to S212. In S212, the image request unit 1104B checks whether the replay condition set in the replay condition setting area 180b3 of the recollection control setting screen 180b is updated. If the replay condition is updated, the process returns to S201 and the process is repeated. If the replay condition is not updated, the process returns to S203 and the process is repeated.

[0134] Figure 7B FIG. 3 shows the image search process of the information server 300. The related information storage area 3012 of the information server 300 stores Fig. 7AThe following processing is mainly controlled by the image search unit 3102.

[0135] In S301, the information server 300 receives the current terminal information (current date and time information, current location information, current orientation information, user identification information) from the portable information terminal 100 (with Fig. 7A The image search unit 3102 obtains the user identification information of the source portable information terminal 100 based on the received current terminal information. Then, the image search condition corresponding to the user identification information among the image search conditions sent in advance from the portable information terminal 100 is confirmed (S302).

[0136] Next, in S303, the image retrieval unit 3102 refers to the associated information database stored in the associated information storage area 3012 to determine whether there is an image that matches the image retrieval condition. This determination is made by comparing the current terminal information (current date and time information, current position information, current orientation information, user identification information) received in S301 with the associated information at the time of shooting registered in the associated information database (date and time information, position information, orientation information, distance information), and referring to the playback conditions recorded in the image retrieval condition. A specific example will be described later. If there is an image that matches the retrieval condition, the process proceeds to S304. If there is no image that matches the retrieval condition, the process ends.

[0137] In S304, it is determined whether the image that matches the search condition (hereinafter referred to as "the image") is an image registered by the user of the portable information terminal 100. In this determination, the user identification information obtained based on the current terminal information and the user identification information of the image data registered in the related information database are compared. As a result of the determination, if the image is an image registered by the user of the portable information terminal 100, the process proceeds to S306. If the image is not image data registered by the user of the portable information terminal 100, the process proceeds to S305.

[0138] In S305, it is determined whether the image data is allowed to be shared with the user identified by the user identification information obtained based on the current terminal information. In this determination, the user identification information obtained based on the current terminal information and the information of the public level setting of the image data in the related information database are compared. As a result of the determination, if the image data is allowed to be shared with the above-identified user, the process proceeds to S306. If the image data is not allowed to be shared with the identified user, the process ends.

[0139] In S306, the image search unit 3102 obtains the image data from the image database in the image data storage area 3011 based on the image recognition information of the image data. Furthermore, the image search unit 3102 sends the obtained image data as a memory image to the portable information terminal 100 together with the related information at the time of shooting registered in the related information database (S307).

[0140] In this way, the information server 300 can search for images matching the search conditions from the image database based on the memory image search conditions set by the portable information terminal 100 , and transmit the images to the portable information terminal 100 as memory images.

[0141] [Specific example of memory image retrieval processing]

[0142] by Figure 6A to Figure 6C The registered data is an example. Figure 7B In the image retrieval process, the current terminal information (current date and time information, current location information, current direction information) acquired from the portable information terminal 100 and the memory image stored in the mobile information terminal 100 are retrieved. Figure 6C The data of the shooting conditions in the related information database is compared with the playback conditions recorded in the memory image retrieval conditions ( Figure 4C ), retrieve the corresponding image.

[0143] <Search Example 1>

[0144] When the current date and time information of the terminal is "October 1, 2017, 13:25:15", and the playback conditions are set to "playback of memory images based on years" and "playback of memory images taken 2 years or more from the current date and time", image data "img_000001" and image data "img_000002" are extracted.

[0145] <Search Example 2>

[0146] When the current date and time information of the terminal is "October 1, 2017, 13:25:15", and the playback conditions are set to "playback of memory images based on years" and "playback of memory images taken 2 years or more ago from the current date and time", and further set to "playback of memory images based on months" and "playback of memory images taken within 1 month before and after", the image data "img_000002" is extracted.

[0147] <Search Example 3>

[0148] When the current date and time information of the terminal is "October 1, 2017, 13:25:15", and the playback conditions are set to "playback of memory images based on time" and "playback of memory images taken within 15 minutes before and after", image data "img_000001" and image data "img_000027" are extracted.

[0149] <Search Example 4>

[0150] When the current location information of the terminal is "35.4030 / 139.5329", the playback conditions are set to "playback of memory images based on the current location" and "playback of memory images taken within an error of latitude and / or longitude of 0.0010", image data "img_000002" and image data "img_000027" are extracted.

[0151] <Search Example 5>

[0152] When the playback conditions are set to "playback of memory images based on designated positions" and "designated position = 35.4050 / 139.5320" and "playback of memory images taken within an error of latitude and / or longitude of 0.0010", image data "img_000003" is extracted regardless of the portable current position information.

[0153] <Search Example 6>

[0154] When the current orientation information of the terminal is "192 degrees" and the playback conditions are set to "playback of memory images based on orientation" and "playback only memory images captured within an orientation difference of ±5 degrees", image data "img_000003" is extracted.

[0155] <Search Example 7>

[0156] When the current date and time information of the terminal is "October 1, 2017 13:25:15", the current location information of the terminal is "35.4030 / 139.5329", the playback conditions are set to "playback of memory images based on years" and "playback of memory images taken 2 years or more ago from the current date and time", set to "playback of memory images based on months" and "playback of memory images taken within 1 month before and after", and further set to "playback of memory images based on current location" and "playback of memory images taken within an error of latitude and / or longitude of 0.0010", the image data "img_000002" is extracted.

[0157] <Search Example 8>

[0158] When the current orientation information of the terminal is "δ degrees" and the playback condition is set to "playback in the direction of the object", the image data is displayed according to Figure 4D The steps shown in the figure calculate the orientation γ of the object based on the current position, and extract the image data when there is image data in which the angle difference between the calculated orientation γ and the current orientation δ is within a predetermined value (X degrees). In addition, in calculating the orientation γ, the current position information and the shooting conditions (position information, orientation information, distance information) registered in the related information database are used.

[0159] <Search Example 9>

[0160] When the playback condition is "playback of a specific user's memory image" and the mail address of "USER000002" is registered, the image data "img_000001" registered by "USER000002" is extracted (when "USER000002" permits "USER000001" to share the memory image).

[0161] In the above search examples, when the current position information of the portable information terminal is used, the playback condition is "playback of the recall image based on the position" (search example 6), and the current position information of the portable information terminal is used (search example 8), which can be said to be particularly unique processing.

[0162] <Display of Memory Image>

[0163] Specifically describe the Fig. 7A The portable information terminal 100 is to display the memory image received from the information server 300, but the information server 300 also sends the associated information when the image was taken. Upon receiving this information, the image display unit 1104C compares the associated information when the displayed image was taken with the current status of the portable information terminal 100 (current terminal information), and performs display control based on the consistency of each information. Several display examples are shown below.

[0164] <Display Example 1>

[0165] Fig. 8AThe display screen corresponding to the search result of the above-mentioned <Search Example 3> is shown. When the current date and time information of the portable information terminal 100 is "October 01, 2017 13:25:15", and the playback condition is "playback of the memory image based on the time", the two extracted image data "img_000001" and "img_000027" and their related information are received from the information server 300. In this case, as shown in image data 180d2 and 180d3, the image display unit 1104C displays the two received image data "img_000001" and "img_000027" by overlapping them with the image data (hereinafter referred to as camera image) 180d1 input from the second image input unit 134.

[0166] In this display, the display sizes of the image data 180d2 and the image data 180d3 are changed, and the image data 180d3 is displayed larger. The current date and time information and the associated information (shooting date and time information) of each image data are compared, and the size is controlled according to the consistency. That is, in this example, the current time information "13:25:15" and the time information of each image data are compared, and the image data 180d3 "img_000027" (13:22:53) with the smaller error is displayed larger, and the other image data 180d2 "img_000001" (13:36:41) is displayed smaller.

[0167] <Display Example 2>

[0168] Figure 8B The display screen corresponding to the search result of the above-mentioned <Search Example 4> is shown. When the current position information of the portable information terminal 100 is "35.4030 / 139.5329" and the playback condition is "playback of the memory image based on the current position", two image data "img_000002" and "img_000027" and their related information are received from the information server 300. In this case, instead of displaying the two image data, the related information (shooting position information) of each image data and the current position information are compared, and the image data with a larger degree of consistency (smaller error) is selected and displayed. In this example, the image data of "img_000027" (shooting position information is "35.4031 / 139.5330") is selected as the image data (memory image) 180e2 and displayed side by side with the image data (camera image) 180e1 input from the second image input unit 134.

[0169] At this time, the display ratio of the image data (camera image) 180e1 and the image data (recall image) 180e2 is changed according to the consistency between the shooting position information of the selected image data "img_000027" and the current position information. Figure 8B In (a), when the error between the two is small (the degree of consistency is large), the display ratio of the image data (recall image) 180e2 is increased. Figure 8B In (b), when the error between the two is large (the degree of coincidence is small), the display ratio of the image data (recall image) 180e2 is reduced.

[0170] Furthermore, when the user moves and updates the current position information of the portable information terminal 100, the display scale is also controlled to be updated according to the current position.

[0171] <Display Example 3>

[0172] Figure 8C The display screen corresponding to the search result of the above-mentioned <Search Example 6> is shown. When the current orientation information of the portable information terminal 100 is "192 degrees" and the playback condition is "playback of the memory image based on the orientation", one image data "img_000003" extracted and its related information are received from the information server 300. In this case, the received image data "img_000003" is displayed as image data (memory image) 180f2, overlapping with the image data (camera image) 180f1 input from the second image input unit 134.

[0173] In addition, the display size of the image data (recall image) 180f2 is controlled based on the consistency between the associated information of the image data (recall image) 180f2 and the current status of the portable information terminal 100 (current date and time information, current location information, current orientation information, etc.). In this example, the shooting orientation information (195 degrees) and the current orientation information (192 degrees) of "img_000003" are compared. Figure 8C In (a), when the degree of coincidence is large (error is small), the image data 180f2 is displayed larger. Figure 8C In (b), when the degree of coincidence is small (the error is large), the image data 180f2 is displayed smaller.

[0174] When displaying the search results of the above-mentioned <Search Example 8>, it can also be applied Figure 8C <Display Example 3>. When the current orientation information of the portable information terminal 100 is "δ degrees" and the playback condition is set to "playback in the direction of the object", image data (recall image) in which the orientation γ of the object is close to the current orientation δ is received. In this case, Figure 8CAs shown, the received image data (recall image) is also displayed as image data 180f2.

[0175] Regarding the size of the image data 180f2 displayed in this case, the orientation γ of the object and the current orientation δ are compared. If the error is small, the image data 180f2 is displayed larger as shown in (a), and if the error is large, the image data 180f2 is displayed smaller as shown in (b). Alternatively, it is also possible to display the image data 180f2 larger if the calculated distance N to the object is small, and display the image data 180f2 smaller if the distance N to the object is large.

[0176] Thus, according to <Display Example 3>, when the current orientation of the portable information terminal 100 owned by the user changes, the size of the displayed memory image changes according to the degree of consistency with the orientation when the image was taken (or the orientation of the object). Furthermore, when the image is facing the orientation when the image was taken (or the orientation of the object), the memory image is displayed at the largest size, which can effectively remind the user of the memory when the image was taken.

[0177] In the above display control, the memory image is displayed with the size changed according to the degree of consistency between the current terminal status and the related information at the time of shooting. However, the image may be displayed with different transparency according to the degree of consistency.

[0178] In the above display example, the memory images being displayed may be controlled to be displayed in full screen when one is selected by a user operation instruction. Alternatively, the display ratio of the selected image and the camera image may be changed.

[0179] In the above-mentioned <Display Examples 1 and 3>, when the portable information terminal 100 is an HMD having a transmissive display, the camera images 180d1 and 180f1 may be the surrounding scenery of the transmissive display. In this case, the memory images 180d2 and 180f2 are displayed superimposed on the surrounding scenery.

[0180] As described above, the portable information terminal 100 of this embodiment obtains images that are closely related to the current terminal status (date, time, location, pointing direction, etc.) from the memory images that the user has taken in the past and registered in the information server 300, and displays them overlapping or side by side with the current surrounding scenery. At this time, the size and transparency of the displayed image are controlled according to the consistency between the current terminal status and the related information at the time of shooting, so that the image that is particularly closely related to the current status is highlighted, which can effectively remind the user of the memory.

[0181] <Viewing device capable of VR display>

[0182] Here, as a usage mode of the portable information terminal 100 , a viewing device that is worn on the head of a user and can display VR (Virtual Reality) or the like will be described.

[0183] Fig. 9A The figures show a state in which a portable information terminal 100 is inserted into a viewing device (adapter) 10 and the viewing device 10 is worn by a user 1. (a) is a top view of the user 1 wearing the viewing device 10 as viewed from directly above, and (b) is a front view of the user 1 wearing the viewing device 10 as viewed from the front. By using such a viewing device 10, the same function as an HMD (Head Mounted Display) can be achieved. Alternatively, the following method can be used: instead of the viewing device 10, the portable information terminal 100 is arranged in front of the user's eyes, and the user 1 holds the portable information terminal 100 with both hands or one hand for use.

[0184] The viewing device 10 is a structure that supports a portable information terminal 100 such as a smartphone or a tablet terminal by inserting it into the slit portion 20. The user 1 places the viewing device 10 with the portable information terminal 100 inserted in front of the eyes and fixes it to the head using a belt or the like. The display screen (display unit 131) of the portable information terminal 100 can perform VR (Virtual Reality) display, MR (Mixed Reality) display, etc., of artificial objects produced using CG (Computer Graphics) or the like superimposed on the image data (camera image) input from the second image input unit 134. In addition, in the case of an HMD with a transmissive display, AR (Augmented Reality) display of artificial objects produced using CG or the like superimposed on the surrounding scenery can be performed.

[0185] The image displayed on the display screen of the portable information terminal 100 enters the eyes of the user 1 through the lens portion 11 disposed inside the viewing device 10. In addition, the spacer 12 prevents the images passing through the left and right lens portions 11 from being mixed. In addition, the above-mentioned slit portion 20, lens portion 11, spacer 12, etc. are for explaining the internal structure of the viewing device 10, and are usually not exposed to the outside. In addition, it is preferable to have the following structure: in normal times, a part of the frame of the viewing device 10 covers the portable information terminal 100 like a cover, so that the portable information terminal 100 can be prevented from falling or shifting even when the head of the user 1 moves up and down or left and right. In addition, it is set to be a structure in which a window 13 is provided in the frame of the viewing device 10 to expose the second image input unit 134 provided on the back of the portable information terminal 100.

[0186] Fig. 9B It is a diagram for explaining the changes in the user's line of sight and the pointing direction of the portable information terminal 100 when the user 1 moves his head. (a) is a top view of the user 1 as viewed from directly above. The user 1 wearing the viewing device 10 moves his head left and right (horizontally) with the center of the head as the axis, and the portable information terminal 100 inserted into the viewing device 10 moves from position 100a to position 100b. In this case, the direction pointed by the portable information terminal 100 at position 100a is Da, and the direction pointed by the portable information terminal 100 at position 100b is Db, and the relative angle difference is δ. In addition, (b) is a diagram showing the change (Da→Db) of the pointing direction of the portable information terminal 100 in (a) by fixing the center position of the portable information terminal 100.

[0187] Here, the use of Fig. 9A , Fig. 9B The viewing device is used to perform the Figure 8C In the case of control of <Display Example 3>. When the direction currently pointed by the portable information terminal 100 is the direction Db and the object is the direction Da, the size of the image data 180f2 is controlled according to the size of the angle difference δ. Furthermore, the transparency of the image data 180f2 can also be changed according to the size of the angle difference δ. In addition, the control is such that when the user 1 moves the head left and right (horizontally) to update the current direction information of the portable information terminal 100, or when the user 1 moves to update the current position information of the portable information terminal 100, the size of the above display is also appropriately updated.

[0188] [Modification 1]

[0189] The image data registered in the image database of the information server 300 may not be the image data that is photographed by the portable information terminal 100 and uploaded via the network 200. For example, a store operator creates an advertising image of his store on a PC and registers it. In this case, in the associated information at the time of shooting registered in the associated information database, the sale day of the store is registered as the shooting date and time information, the latitude / longitude information in front of the front entrance of the store is registered as the shooting location information, and the direction information when pointing from the position in front of the front entrance of the store to the front entrance of the store is registered as the shooting direction information. In addition, as a setting for the "Shared Image" item of the public level setting, select "Share with all users".

[0190] Therefore, when any user points the portable information terminal 100 at the front entrance of the store operated by the store operator on a day when the "day" information coincides with the special sale day, the portable information terminal 100 can obtain and display the advertisement image from the information server 300. That is, the store operator can effectively promote the store by preparing the advertisement image in advance and registering it as a recall image.

[0191] [Modification 2]

[0192] When the image data captured by the portable information terminal 100 and its related information are registered in the information server 300 , other related data (such as voice data, text data, etc.) are also registered. The related data is registered in the related file database (not shown) of the information server 300 .

[0193] For example, in the above-mentioned recall image playback process ( Fig. 7A ) receives the image data (memory image) from the server 300 (S208), and simultaneously receives the associated data. Thereafter, when the image data (memory image) is displayed on the display unit 131 (S210), the associated data is simultaneously replayed. If it is voice data, the voice replay software is started and the voice is output from the speaker, and if it is text data, it is displayed on the display unit 131 through a notepad APP or the like. Alternatively, in the process of S208, only the associated data may be acquired, and after the image data (memory image) displayed on the display unit 131 is selected through a user operation instruction, the acquired associated data may be replayed. By performing such control, the data associated with the image can be replayed while the image data (memory image) is displayed, so that memories can be more effectively recalled.

[0194] The above describes an example of an embodiment of the present invention, but the structure of the technology for implementing the present invention is certainly not limited to the above-mentioned embodiment, and various modifications can be considered. For example, a part of the structure of a certain embodiment can be replaced with the structure of other embodiments, and the structure of other embodiments can be added to the structure of a certain embodiment. All of these structures belong to the scope of the present invention. In addition, the numerical values, messages, etc. that appear in the text and the figure are only examples, and even if different numerical values ​​or messages are used, the effect of the present invention will not be affected.

[0195] Part or all of the functional modules of the present invention described above may also be implemented in hardware by, for example, implementing them with integrated circuits. In addition, they may also be implemented in software by interpreting and executing programs for implementing various functions by a microprocessor unit or the like. Hardware and software may also be used in combination. The software may also be in a state pre-stored in the ROM 103, the storage device unit 110, etc. of the portable information terminal 100 at the time when the product is shipped. It may also be obtained from various server devices on the Internet after the product is shipped. In addition, the software provided by a memory card, a CD, etc. may also be obtained.

[0196] In addition, the control lines and information lines shown in the drawings are those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines on the product. It can also be considered that almost all the structures are actually connected to each other.

Claims

1. A mobile information terminal, comprising: A camera is configured to output an image signal; an image signal processor configured to generate captured image data based on the image signal; A GPS signal receiver configured to receive a GPS signal; wireless communication circuitry configured to communicate with a network; a display configured to display an image from the captured image data; A memory configured to store the captured image data; as well as The control circuit is configured as follows: using the received GPS signal to obtain location information; causing the wireless communication circuit to send at least the captured image data and the location information to the network; causing the wireless communication circuit to receive stored image data and stored information via the network, wherein the stored image data and stored information are stored in an external server and are related to at least the captured image data or the location information; as well as displaying on a display both an image from the captured image data and an image from the stored image data, Wherein, the control circuit is further configured to change a display size of the stored image data received from the external server based on a predetermined condition.

2. The mobile information terminal according to claim 1, further comprising a geomagnetic sensor, wherein the predetermined condition is related to the direction angle of the mobile information terminal, and The control circuit is further configured to obtain the direction angle using a signal from the geomagnetic sensor.

3. The mobile information terminal according to claim 1, further comprising an infrared rangefinder for measuring the distance between an object and the mobile information terminal, wherein the predetermined condition is related to distance information, and in, The control circuit is further configured to obtain the distance information using a signal from the infrared range finder.

4. The mobile information terminal according to claim 1, wherein the predetermined condition is related to current location information, and the current location information is update information of the location information. 5 . The mobile information terminal according to claim 1 , wherein the memory is configured to store the captured image in the form of Exchangeable Image File Format (EXIF).

6. The mobile information terminal according to claim 1, further comprising a touch panel, wherein the touch panel is configured to receive user operations, in, The control circuit is configured to switch a display size of captured image data and a display size of stored image data to each other when receiving the user operation from the touch panel.

7. The mobile information terminal according to claim 1, wherein the stored information is direction angle information indicating a direction angle of the mobile information terminal.

8. The mobile information terminal according to claim 1, wherein: The stored information relates to distance information indicating the distance between the object and the mobile information terminal.

9. The mobile information terminal according to claim 1, wherein: The stored information is related to location information indicating the location of the mobile information terminal.

10. A mobile information terminal, comprising: A camera is configured to output an image signal; an image signal processor configured to generate captured image data based on the image signal; A GPS signal receiver configured to receive a GPS signal; wireless communication circuitry configured to communicate with a network; a display configured to display an image from the captured image data; A touch panel configured to receive user operations; A memory configured to store the captured image data; as well as The control circuit is configured as follows: obtaining location information using the received GPS signal and / or the signal received by the wireless communication circuit; causing the wireless communication circuit to send at least the captured image data and the location information to the network; causing the wireless communication circuit to receive stored image data and stored information via the network, wherein the stored image data and stored information are stored in an external server and are related to at least the captured image data or the location information; as well as displaying on a display both an image from the captured image data and an image from the stored image data, Wherein, the control circuit is further configured as: changing the display size of the stored image data based on a predetermined condition, and When a user operation is received from the touch panel, the display size of the captured image data and the display size of the stored image data are switched to each other.

11. A method for controlling a mobile information terminal, the mobile information terminal comprising a camera and a display, the method comprising: generating captured image data based on an image signal from the camera; Receive GPS signals; Storing the captured image data in a memory; obtaining location information using the received GPS signal; Sending at least the captured image data and the location information to a network; receiving stored image data and stored information via the network, wherein the stored image data and the stored information are stored in an external server and are at least related to the captured image data or the location information; displaying on a display both an image from the captured image data and an image from the stored image data; as well as A display size of the stored image data from the external server is changed based on a predetermined condition.

12. The method according to claim 11, The mobile information terminal further includes a geomagnetic sensor. wherein the predetermined condition is related to the direction angle of the mobile information terminal, and The method further comprises obtaining the direction angle using a signal from the geomagnetic sensor.

13. The method according to claim 11, in, The mobile information terminal further comprises an infrared rangefinder for measuring the distance between the object and the mobile information terminal. wherein the predetermined condition is related to distance information, and The method further comprises obtaining the distance information using a signal from the infrared rangefinder.

14. The method according to claim 11, wherein: The predetermined condition is related to current position information as update information of the position information.

15. The method according to claim 11, wherein: The storing includes storing the captured images in the form of Exchangeable Image File Format (EXIF).

16. The method according to claim 11, characterized in that in, The mobile information terminal further includes a touch panel configured to receive user operations, and The method further includes switching a display size of the captured image data and a display size of the stored image data to each other when a user operation is received from the touch panel.

17. The method according to claim 11, wherein: The stored information is direction angle information indicating the direction angle of the mobile information terminal.

18. The method according to claim 11, wherein: The stored information relates to distance information indicating the distance between the object and the mobile information terminal.

19. The method according to claim 11, wherein: The stored information is related to location information indicating the location of the mobile information terminal.

Images