system

Abstract

Provide a system. 【Solution means】 Means for collecting past image data, Means for extracting metadata from the image data, Means for adding location information based on the metadata, Means for storing the location information - attached image data in a database, Means for receiving a location specification from a user, Means for searching the database based on the location specification, Means for providing past image data as a search result, A system including means for integrating and displaying the past image data and current map data.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, there has been an increasing need for real estate agents and home buyers to refer to past land use situations and historical changes. However, in the previous information provision services, it has been difficult to efficiently acquire past images and photos and integrate them with modern map data for display. In addition, there is also a problem that past sensitive information is not appropriately processed, which may cause privacy issues and misunderstandings. The present invention aims to solve these problems and provides a system for efficiently linking and displaying past image data and current map data.

Means for Solving the Problems

[0005] The present invention solves the aforementioned problems with a system that includes means for collecting historical image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving location specifications from a user, means for searching the database based on the location specifications, means for providing historical image data as a search result, and means for integrating and displaying the historical image data and current map data. Furthermore, the present invention achieves more efficient and secure information provision by including means for automatically acquiring image data from publicly available databases on the internet as the image data collection means, means for analyzing information regarding the date of shooting, location of shooting, and events in the image data using artificial intelligence technology as the metadata extraction means, means for creating an index to improve search efficiency as the location-attached image data is stored in a database as the search result, and means for filtering out sensitive information in the historical image data provided as a search result.

[0006] "Historical image data" refers to image and photographic data associated with a specific past date, time, and location, collected from publicly available databases on the internet and other sources.

[0007] "Metadata" refers to supplementary data extracted from past image data, including the date, location, and events associated with the image.

[0008] "Location information" refers to latitude and longitude data that indicates the location where a particular image data was taken.

[0009] A "database" is an information management system that systematically stores historical image data, metadata, and location information, making it searchable and accessible.

[0010] "Location specification" refers to location information that a user enters to indicate a specific geographical location from which they want to refer to past information.

[0011] A "search method" refers to a function or module for searching information within a database and retrieving historical image data related to a specified location.

[0012] A "user" is a person who uses this system through a dedicated application or website to specify a location or view past image data.

[0013] "Current map data" refers to map data that shows current geographical information, and is displayed by integrating it with historical image data.

[0014] "Artificial intelligence technology" is a technology in which computers automatically analyze data and extract metadata such as the date and location of the photograph, and events that occurred.

[0015] An "index" is a specific search structure created to efficiently retrieve information within a database.

[0016] "Sensitive information" refers to delicate information that requires caution when being made public, such as information related to an individual's privacy or detailed information about disaster-stricken areas.

[0017] "Filtering" is the process of modifying or removing data in order to prevent sensitive information from being displayed. [Brief explanation of the drawing]

[0018] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0020] First, the language used in the following description will be explained.

[0021] In the following embodiments, the signed processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Furthermore, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), and APU (Accelerated Processing Unit).

[0022] In the following embodiments, signed RAM (Random Access Memory) is a memory that temporarily stores information and is used as work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] The present invention is a system that collects historical image data, organizes it based on metadata, adds location information and stores it in a database, searches based on the user's specified location, and displays the integrated historical image data and current map data. The following describes in detail how the present invention is implemented.

[0040] System Configuration

[0041] 1. Server: This server is central to the present invention, performing data processing and storage, and responding to user requests.

[0042] 2. Device: A device (such as a smartphone, tablet, or PC) that a user uses to access and operate the system through an application or website.

[0043] 3. User: A person who specifies a location, views the results, and performs operations through a dedicated interface.

[0044] System operation

[0045] First, historical image data is collected. The server executes a script that automatically retrieves image data from publicly available databases on the internet. The collected image data is temporarily stored in storage.

[0046] Next, the server analyzes the collected image data using AI image recognition technology. The AI ​​extracts metadata from the images, including the date, location, and events that occurred. Based on this metadata, the server adds location information (latitude and longitude) to each image.

[0047] The image data with added location information is stored in a specialized database. This database is indexed later to enable efficient searching.

[0048] Users access the system through a dedicated application or website. Users specify a location they want to view on a map and submit a request. The server receives this request and searches its database for historical image data corresponding to the specified location.

[0049] The server organizes the search results and sends them to the user's terminal. The terminal integrates the received historical image data and metadata with current map data and displays it on the screen. This allows the user to compare and verify past and present conditions.

[0050] Specific example

[0051] For example, consider a case where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. This information is sent to the server by the user's action. The server searches its database for entries containing historical image data related to the specified location in Shinjuku Ward, and retrieves relevant data such as "a photograph of a residential area in Shinjuku Ward in the 1930s" or "a depiction of the area during the reconstruction period after the Great Kanto Earthquake."

[0052] The server filters this data, removing sensitive information (such as detailed addresses or images of individuals' faces), and then sends the organized data to the user's device. The device receives the data and displays past images and their descriptions on the screen along with a current map. The user can easily compare the past and present.

[0053] Thus, the present invention is a system that efficiently links historical image data and current map data to provide useful information to users.

[0054] The following describes the processing flow.

[0055] Step 1:

[0056] Server: Executes scripts to access publicly available databases and libraries on the internet and collect historical image data. The collected image data is stored in temporary storage.

[0057] Step 2:

[0058] Server: The server initiates analysis of the collected image data using AI image recognition technology. The AI ​​model extracts metadata from the images, including the date, location, and events that occurred.

[0059] Step 3:

[0060] Server: Based on the extracted metadata, it adds location information such as latitude and longitude to each image. This information is used to set location attributes for the image data.

[0061] Step 4:

[0062] Server: Stores location-tagged image data in a database. An index is created during storage to enable efficient searching later.

[0063] Step 5:

[0064] User: Access the dedicated application or website and log in. The user uses the map interface to specify the location for which they want to view past information by clicking or tapping.

[0065] Step 6:

[0066] Terminal: Sends the user's location information as a request to the server. The request includes the latitude and longitude information of the specified location.

[0067] Step 7:

[0068] Server: Based on the received location request, it searches the database. It retrieves historical image data and its metadata associated with the specified latitude and longitude.

[0069] Step 8:

[0070] Server: Organizes search results and prepares image datasets related to the specified locations. Filters out sensitive information to ensure it is appropriate for publication.

[0071] Step 9:

[0072] Server: Sends filtered data to the user's terminal. The data is sent in a structured format such as JSON.

[0073] Step 10:

[0074] Terminal: Analyzes received data and integrates historical image data and current map data on the interface for display. Users can simultaneously view historical and current information for a specified location.

[0075] Step 11:

[0076] User: Based on the displayed information, users can view more detailed information as needed or based on their interests. Users can also specify other locations and check past information using a similar procedure.

[0077] This series of processes allows users to view historical image data and current map data in an integrated manner, making it easy to understand the historical background and past conditions of a location.

[0078] (Example 1)

[0079] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0080] Conventional technologies had problems with efficiency and accuracy in systems that properly collected, analyzed, and stored historical image data and linked it with location information for searching. Furthermore, the process of integrating historical image data corresponding to a user-specified region with current map data presented challenges in filtering search results and efficiently managing the data.

[0081] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0082] In this invention, the server includes means for collecting historical image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving a location specification from a user, means for searching the database based on the location specification, means for providing historical image data as search results, means for integrating and displaying the historical image data and current map data, means for filtering the search results, and means for performing indexing. This makes it possible to efficiently and accurately collect and store historical image data and provide appropriate search results in response to user requests.

[0083] "Image data" refers to digital image information that records past events and places.

[0084] "Metadata" refers to information associated with the image data itself, including data such as the date and time of shooting, the location of shooting, and details of the event.

[0085] "Location information" refers to digital data of latitude and longitude used to indicate a specific location.

[0086] A "database" is an information management system for efficiently storing, managing, and retrieving collected image data and metadata.

[0087] "Location specification" refers to the geographical location that a user specifies when they wish to obtain information about a particular place.

[0088] "Search results" refer to information extracted from a database of past image data related to the location specified by the user.

[0089] "Filtering" is the process of removing unnecessary data or sensitive information from search results.

[0090] "Indexing" is the process of organizing and indexing data stored in a database so that it can be searched efficiently.

[0091] A "user" is an individual or group that performs operations to obtain information using a system.

[0092] A "server" is a central management system that collects, analyzes, stores, searches, and provides results to users for image data.

[0093] A "terminal" is a device (such as a smartphone, tablet, or PC) that a user uses to access and operate a system.

[0094] The present invention is a system that collects historical image data, organizes it based on metadata, adds location information and stores it in a database, searches based on the user's specified location, and displays the integrated historical image data and current map data. The following describes in detail how the present invention is implemented.

[0095] First, historical image data is collected. The server executes a script that automatically retrieves image data from publicly available databases on the internet. For example, the "Public Image Database API" or the "Image Search API" can be used. The collected image data is temporarily stored in storage (for example, Amazon S3).

[0096] Next, the server analyzes the collected image data using AI image recognition technology (for example, Google® Cloud Vision API). Here, metadata such as the date and location of the image, and information about the event are extracted from the image data. Based on this metadata, the server adds location information (latitude and longitude) to each image. Specifically, location information is obtained using the Google Maps Geocoding API.

[0097] The added location-tagged image data is stored in a database (e.g., PostgreSQL). This database is indexed using Elasticsearch® to enable efficient searching later on.

[0098] Users access the system through a dedicated application or website. Users specify a location they want to view on a map and submit a request. This request is received by the server, which searches its database for historical image data corresponding to the specified location. The server then organizes the search results, excluding sensitive information (such as detailed addresses or images of individuals' faces) before sending them to the user's device.

[0099] The device integrates received historical image data and metadata with current map data (e.g., Google Maps API) and displays it on the screen. This allows the user to compare and verify past and present conditions.

[0100] Specific example

[0101] For example, consider a case where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. This information is sent to the server as a result of the user's actions. The server searches its database for entries containing historical image data related to the specified location in Shinjuku Ward and retrieves relevant data such as "a photograph of a residential area in Shinjuku Ward in the 1930s" or "a picture of the area during the disaster recovery period."

[0102] The server filters and organizes this data, then sends the organized data to the user's terminal. The terminal receives the data and displays past images and their descriptions on the screen along with the current map. The user can easily compare the past and the present.

[0103] Examples of prompts to input into a generative AI model

[0104] Example of a prompt:

[0105] Please provide a detailed explanation of how a system works to integrate "photographs of residential areas in Shinjuku Ward from the 1930s" with "a current map of Shinjuku Ward."

[0106] Please describe the specific steps involved in each stage of the process.

[0107] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0108] Step 1:

[0109] The server automatically retrieves image data from publicly available databases on the internet.

[0110] Input: API endpoint for the public image database.

[0111] Data processing / calculation: Send an API request and temporarily store the retrieved image data.

[0112] Output: Temporarily saved image data.

[0113] Specific operation: A Python script is executed to send requests to a public database API and save the retrieved image data to storage such as Amazon S3.

[0114] Step 2:

[0115] The server analyzes the collected image data using AI image recognition technology.

[0116] Input: Temporarily saved image data.

[0117] Data processing / calculation: Send images to an AI image recognition service and extract metadata such as the date and location of the photo, and information about the event.

[0118] Output: Image data with metadata attached.

[0119] Specific operation: Use the Google Cloud Vision API to analyze image data and retrieve metadata including the date and time and location of the image.

[0120] Step 3:

[0121] The server adds latitude and longitude location information to the image based on the metadata.

[0122] Input: Image data with metadata attached.

[0123] Data processing / calculation: Use the Geocoding API to convert text information (address / place name) into latitude and longitude, and add location information to image data.

[0124] Output: Image data with latitude and longitude information added.

[0125] Specific operation: Use the Google Maps Geocoding API to parse the address information in the metadata and obtain the appropriate latitude and longitude location information.

[0126] Step 4:

[0127] The server stores location-tagged image data in a database.

[0128] Input: Image data with latitude and longitude information attached.

[0129] Data processing / calculation: Insert image data into the database and create indexes to enable efficient searching.

[0130] Output: Image data with location information stored in the database.

[0131] Specific operation: Store image data in a PostgreSQL database and perform indexing using Elasticsearch.

[0132] Step 5:

[0133] Users access the system through a dedicated application or website, specify the location they want to view on a map, and submit a request.

[0134] Input: Geographic location information specified by the user.

[0135] Data processing / calculation: Send a request to the server and query information from the specified location.

[0136] Output: Request received by the server.

[0137] Specific operation: The user uses an application or website to click on a location they want to see on a map and sends that information to the server.

[0138] Step 6:

[0139] The server receives a user request and searches the database for historical image data corresponding to the specified location.

[0140] Input: Location request received from the user.

[0141] Data processing / calculation: Search the database for image data corresponding to the queried location and perform filtering.

[0142] Output: Filtered search results.

[0143] Specific operation: Execute Elasticsearch queries to search the database for appropriate image data and exclude sensitive information.

[0144] Step 7:

[0145] The server organizes the search results and sends them to the user's device.

[0146] Input: Filtered search results.

[0147] Data processing / calculation: Organize search results and convert them into a format that can be sent to the user.

[0148] Output: The organized data sent to the user's terminal.

[0149] Specific operation: Convert the search results to JSON format and send them to the user's terminal as an HTTP response.

[0150] Step 8:

[0151] The terminal integrates the received historical image data and metadata with current map data and displays it on the screen.

[0152] Input: Data sent from the server.

[0153] Data Processing / Calculation: Process the received data into a format that can be displayed by integrating it with the map API.

[0154] Output: Integrated data displayed on the user's screen.

[0155] Specific operation: Use the Google Maps API to integrate historical image data with current map data and display it on the user's device.

[0156] (Application Example 1)

[0157] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0158] Conventional technologies had limited means of collecting and effectively providing historical image data to users, making it difficult to compare past and present situations. Furthermore, there was no system that could provide users with relevant contemporary product information based on their interests and enable purchases. Therefore, there is a need to improve the convenience for users when purchasing related products using historical information.

[0159] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0160] In this invention, the server includes means for collecting past image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving a location specification from a user, means for searching the database based on the location specification, means for providing past image data as a search result, means for integrating and displaying the past image data and current map data, means for displaying past and current image data and providing explanatory text, and means for providing related product information based on the user's interests and enabling purchase. This makes it possible for the user to easily compare past and present situations and effectively purchase related products.

[0161] "Past image data" refers to image data taken at a specific time, representing past events or scenes.

[0162] "Metadata" refers to attribute information associated with image data, including information such as the date and time of shooting, location, and events.

[0163] "Location information" refers to geographical information such as latitude and longitude that indicates the location where the image data was taken.

[0164] A "database" is a system that systematically stores and manages digital data, enabling efficient searching.

[0165] "Location specification" is the operation of specifying a particular geographical location that the user is interested in.

[0166] "Searching" is the operation of finding data within a database based on specific criteria.

[0167] "Current map data" refers to digital map data that displays the latest geographical information.

[0168] "Integrated display" means displaying different data together on a single screen or in a single format.

[0169] "Displaying past and present image data" means displaying past image data and current map data on the screen simultaneously.

[0170] "Description" refers to additional information or explanatory text related to image data.

[0171] "Related product information" refers to product data provided based on the user's interests.

[0172] "Means of enabling purchase" refers to a function that allows users to directly purchase products they are interested in.

[0173] This invention is a system that collects historical image data, organizes it based on metadata, adds location information, stores it in a database, searches based on the user's specified location, and displays the integrated historical image data and current map data. Furthermore, it includes a function to provide relevant product information based on the user's interests and enable purchases.

[0174] The server is at the heart of the system and performs the following roles:

[0175] 1. Server

[0176] The server collects historical image data, extracts metadata, adds location information, and stores it in a database. Furthermore, it receives location information from the user, searches the database based on that information, and provides historical image data as search results. The server integrates current map data with historical image data and displays it, providing the user with relevant product information.

[0177] 2. Terminal

[0178] A terminal is a device that users access and operate through a dedicated application. This includes smartphones, tablets, and PCs. The terminal displays historical image data and current map data received from the server, and displays explanatory text and related product information according to user requests.

[0179] 3. User

[0180] Users specify their location through a dedicated interface and view and manipulate data provided by the system. They can compare past and present situations, obtain relevant product information, and purchase items of interest directly.

[0181] Specific example

[0182] The following is a specific example of the process.

[0183] The user launches the "Time Travel Shopping App" on their smartphone and specifies a particular location on the map, for example, "Shinjuku Ward." They also enter a time period for which they want to view past images, such as "1930-1940." This information is sent from the device to the server. The server searches its database based on the specified latitude, longitude, and date range and retrieves the corresponding past image data.

[0184] The acquired image data includes metadata such as the date and location of the photograph, and this information is transmitted to the device along with the image data. The device integrates the past image data with current map data and displays the past and present images simultaneously. A descriptive text related to the image is also displayed.

[0185] Furthermore, the server provides relevant product information based on the user's interests. For example, if an image of a shopping street from the 1930s is displayed, products that were sold at that time and related modern items will be introduced. Users can purchase products they are interested in directly within the app.

[0186] Example of a prompt

[0187] You are an assistant that searches historical image data and integrates it with current map data for display. When given a location name and date range, search for historical image data of that location and display it compared to current map data. Process the following requests:

[0188] Location: Shinjuku Ward

[0189] Start date: 1930-01-01

[0190] End date: 1940-12-31

[0191] This invention allows users to easily access historical information and compare it with the present to gain a deeper historical understanding. Furthermore, by providing related product information, users can proceed directly to the purchase process, significantly improving convenience.

[0192] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0193] Step 1:

[0194] The user specifies a location and time period on a map.

[0195] The user launches the "Time Travel Shopping App" on their smartphone, specifies a location of interest on the map, and enters the period (start and end dates) for which they want to view past images. The entered information is then sent from the device to the server.

[0196] input

[0197] Information about the specified location (e.g., Shinjuku Ward)

[0198] Specified period (start date, end date)

[0199] output

[0200] Request data to the server (location information and time period information)

[0201] Specific actions

[0202] The terminal generates an API request that sends the specified location and time period to the server via the user interface.

[0203] Step 2:

[0204] The server searches for past image data.

[0205] The server searches the database based on the received request data (location information and time period information) to retrieve relevant historical image data. The server uses indexing to perform efficient searches based on location information and time period.

[0206] input

[0207] Request data (location information and time period information)

[0208] output

[0209] Search results (corresponding past image data)

[0210] Specific actions

[0211] The server executes database queries based on the specified latitude, longitude, and date ranges and collects the corresponding image data.

[0212] Step 3:

[0213] The server analyzes the image data and extracts the metadata.

[0214] The server analyzes the acquired image data using AI image recognition technology and extracts metadata for each image, including the date and location of the image, as well as information about related events.

[0215] input

[0216] Search results (past image data)

[0217] output

[0218] Image data with metadata

[0219] Specific actions

[0220] The server uses an AI model (e.g., an image recognition model) to analyze image data and automatically extract metadata such as the date and location of the photograph, and information about the event.

[0221] Step 4:

[0222] The server adds location information and stores it in the database.

[0223] The server adds location information (latitude and longitude) to each image data based on the extracted metadata and stores it in the database as image data with location information.

[0224] input

[0225] Image data with metadata

[0226] output

[0227] Image data with location information

[0228] Specific actions

[0229] The server calculates the latitude and longitude information corresponding to each image based on the metadata, adds location information, and stores it in the database.

[0230] Step 5:

[0231] The server integrates current map data and historical image data and transmits it.

[0232] The server integrates location-based image data with current map data, converts it into a data format that allows the user to compare past images with the current map, and sends it to the terminal.

[0233] input

[0234] Image data with location information

[0235] Current map data

[0236] output

[0237] Integrated data (data from past images and current maps)

[0238] Specific actions

[0239] The server overlays historical images onto map data based on location information and converts them into a format that allows for comparison and display of both.

[0240] Step 6:

[0241] The device displays integrated data and provides explanatory text.

[0242] The device displays the received integrated data and compares past and present images. It also displays explanatory text related to the images.

[0243] input

[0244] Integrated data (data from past images and current maps)

[0245] output

[0246] Images and descriptions displayed on the user interface

[0247] Specific actions

[0248] The device displays integrated data in a user-friendly format and provides a convenient interface for comparing past and present situations.

[0249] Step 7:

[0250] The server provides relevant product information based on the user's interests.

[0251] The server generates relevant product information based on past image data, descriptions, and user operation data, and transmits this information to the terminal.

[0252] input

[0253] User operation data

[0254] Past image data

[0255] Description

[0256] output

[0257] Related product information

[0258] Specific actions

[0259] The server uses a generative AI model to analyze user interests and generate data to select and provide corresponding product information.

[0260] Step 8:

[0261] The device displays related product information and enables the purchase process.

[0262] The terminal displays the received related product information, allowing the user to purchase items they are interested in directly.

[0263] input

[0264] Related product information

[0265] output

[0266] Product information and purchase options displayed on the user interface

[0267] Specific actions

[0268] The terminal displays related product information and provides an interface that allows users to proceed directly to the purchase process by clicking on a product.

[0269] The above outlines the specific processing steps. This processing flow allows users to easily compare past and current situations and gain an effective experience in purchasing related products.

[0270] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0271] This invention combines a system that collects historical image data, organizes it based on metadata, stores it in a database with location information, searches it based on the user's specified location, and displays the integrated historical image data and current map data with an emotion engine that recognizes the user's emotions. This enables the provision of optimal information according to the user's emotional state.

[0272] System Configuration

[0273] 1. Server: This server is central to the present invention, performing data processing and storage, and responding to user requests. It also incorporates an emotion engine to analyze user emotions.

[0274] 2. Device: A device (such as a smartphone, tablet, or PC) that a user uses to access and operate the system through an application or website.

[0275] 3. User: A person who specifies a location, views the results, and performs operations through a dedicated interface.

[0276] System operation

[0277] First, historical image data is collected. The server executes a script that automatically retrieves image data from publicly available databases on the internet. The collected image data is temporarily stored in storage.

[0278] Next, the server analyzes the collected image data using AI image recognition technology. The AI ​​extracts metadata from the images, including the date, location, and events that occurred. Based on this metadata, the server adds location information (latitude and longitude) to each image.

[0279] The image data with added location information is stored in a specialized database. This database is indexed later to enable efficient searching.

[0280] The user accesses the system through a dedicated application or website. The user specifies the location they want to view on the map and sends a request. This request is received by the server, which searches the database for past image data corresponding to the specified location.

[0281] The server organizes the search results and prepares an image data set related to the specified location. It filters out sensitive information included to check whether it is appropriate for public release.

[0282] Here, an emotion engine is added. The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions in real time. Emotions are classified into categories such as positive, negative, and neutral.

[0283] Based on the user's emotional state, the server optimizes the information to be provided. For example, when the user shows negative emotions, it performs processing such as preferentially providing image data with generally positive content. It also performs filtering corresponding to a specific emotional state to prevent the display of information that is unpleasant to the user.

[0284] Finally, the server sends the filtered data to the user's terminal. The terminal analyzes the received data and integrates and displays the past image data and the current map data on the interface. The user can view the past and current information of the specified location simultaneously.

[0285] Specific Example

[0286] For example, consider the case where the user specifies a residential area in Shinjuku Ward. The user launches the application and selects a point in Shinjuku Ward on the map. On the other hand, the emotion engine recognizes "positive emotions" from the user's facial expressions and voice. This information is sent to the server.

[0287] The server searches the database for entries containing historical image data related to a specified location in Shinjuku Ward, retrieving entries such as "a photograph of a residential area in Shinjuku Ward in the 1930s" or "positive anecdotes from the reconstruction period after the Great Kanto Earthquake." Simultaneously, the sentiment engine optimizes the data to provide an overall positive content, taking into account the user's positive emotions.

[0288] The server filters this data, excluding sensitive information, and then sends the organized data to the user's device. The device receives the data and displays past images and their descriptions on the screen along with a current map. Users can easily compare the past and present while prioritizing the viewing of positive information.

[0289] Thus, the present invention is a system that efficiently links past image data and current map data, and further utilizes an emotion engine to provide optimal information tailored to the user's emotional state.

[0290] The following describes the processing flow.

[0291] Step 1:

[0292] Server: Executes scripts to access publicly available databases and libraries on the internet and collect historical image data. The collected image data is stored in temporary storage.

[0293] Step 2:

[0294] Server: The server initiates analysis of the collected image data using AI image recognition technology. The AI ​​model extracts metadata from the images, including the date, location, and events that occurred.

[0295] Step 3:

[0296] Server: Based on the extracted metadata, add location information such as latitude and longitude corresponding to each image. Using this information, set the positional attributes for the image data.

[0297] Step 4:

[0298] Server: Save the image data with location information to the database. Create an index during saving to enable efficient subsequent searches.

[0299] Step 5:

[0300] User: Access and log in to a dedicated application or website. The user uses the map interface to specify the location where they want to view past information by clicking or tapping.

[0301] Step 6:

[0302] Terminal: Send the user's location specification information to the server as a request. The request includes the latitude and longitude information of the specified location.

[0303] Step 7:

[0304] Server: Search the database based on the received location request. Retrieve the past image data and its metadata related to the specified latitude and longitude.

[0305] Step 8:

[0306] Server: Organize the search results and prepare an image data set related to the specified location. Filter the included sensitive information to check whether it is appropriate for public release.

[0307] Step 9:

[0308] Server: Send the filtered data to the user's terminal. The data is sent in a structured format such as JSON.

[0309] Step 10:

[0310] Terminal: Analyzes received data and integrates historical image data and current map data on the interface for display. Users can simultaneously view historical and current information for a specified location.

[0311] Step 11:

[0312] User: Uses the device's camera and microphone to express emotions through facial expressions and voice. The emotion engine acquires this data and analyzes emotions in real time.

[0313] Step 12:

[0314] Server: The emotion engine classifies the user's emotions as positive, negative, or neutral, and provides optimized information based on that emotional state.

[0315] Step 13:

[0316] Server: Filters relevant metadata based on emotional state. For example, if a user is expressing negative emotions, it prioritizes providing image data with positive content.

[0317] Step 14:

[0318] Server: Retransmits the information set optimized by the emotion engine to the user's terminal.

[0319] Step 15:

[0320] Device: Redisplays past images and their descriptions that match the user's emotional state, along with the current map. Users can enjoy viewing information in an emotionally sensitive way.

[0321] This series of steps allows users to view historical image data and current map data in an integrated manner, and then receive optimized information from the emotion engine.

[0322] (Example 2)

[0323] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0324] Conventional systems that collect historical image data, add location information, store it in a database, and search based on the user's specified location do not consider the user's emotional state, and the information provided may not always be optimal. Furthermore, if sensitive information is not filtered, inappropriate information may be provided. To solve this problem, a system is needed that provides optimal information tailored to the user's emotional state.

[0325] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting past image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the image data with location information in a database, means for receiving a location specification from the user, means for searching the database based on the location specification, means for providing past image data as a search result, means for integrating and displaying past image data and current map data, and an emotion engine that recognizes the user's emotions, analyzes the user's emotions using the emotion engine, and optimizes the information to be provided based on the analysis results. This makes it possible to provide optimal information according to the user's emotional state.

[0326] "Past image data" refers to photographs and image files taken in the past, including data such as the date, time, and location of the shooting.

[0327] "Metadata" refers to supplementary information related to image data, including data such as the date and time of shooting, location, and details of the event.

[0328] "Location information" refers to longitude and latitude data used to indicate a specific location.

[0329] A "database" refers to a specialized data storage system for efficiently storing, searching, and managing image data, its metadata, and location information.

[0330] A "user" refers to a person who specifies a location or performs other operations through the system.

[0331] "Location specification" refers to the user selecting a location they want to see on a map or specifying it in text.

[0332] An "emotion engine" refers to a system equipped with technology that analyzes a user's emotions in real time from their facial expressions and voice, and classifies that emotional state as positive, negative, neutral, etc.

[0333] "Integration" refers to combining historical image data and current map data and displaying them on a single interface.

[0334] "Optimization" refers to adjusting the content and order of information provided based on the user's emotional state, as analyzed by the emotion engine, to make the user experience as effective as possible.

[0335] "Sensitive information" refers to information that should be restricted from public disclosure based on specific criteria, such as personal information or data deemed socially inappropriate.

[0336] Modes for carrying out the invention

[0337] This invention combines a system that collects historical image data, organizes it based on metadata, stores it in a database with location information, searches it based on the user's specified location, and displays the integrated historical image data and current map data with an emotion engine that recognizes the user's emotions. This enables the provision of optimal information according to the user's emotional state.

[0338] Hardware and software configuration

[0339] 1. Server:

[0340] The server performs the central data processing and storage of this invention, as well as responding to user requests. It also incorporates an emotion engine to analyze user emotions. This requires a high-performance server and utilizes software such as Python, MySQL®, and OpenCV.

[0341] 2. Terminal:

[0342] A terminal is a device that a user uses to access and operate a system through an application or website. Typically, smartphones, tablets, and PCs are used.

[0343] 3. User:

[0344] The user is a person who specifies a location, views the results, and performs actions through a dedicated interface.

[0345] Program Processing Description

[0346] First, historical image data is collected. The server executes a script to automatically retrieve image data from publicly available databases on the internet. The collected image data is temporarily stored in the server's storage. Next, the server analyzes the collected image data using AI image recognition technology (e.g., OpenCV models). The AI ​​extracts metadata from the images, such as the date and location of the image, and events, and adds location information (latitude and longitude) based on this metadata. The image data with location information is stored in a specialized database, such as a MySQL database, and indexed to enable efficient searching.

[0347] Users access the system through a dedicated application or website. Users specify a location they wish to view on a map and submit a request. The server receives this request and searches its database for historical image data corresponding to the specified location. The server organizes the search results and prepares an image dataset related to the specified location. During this process, it filters out sensitive information and verifies whether the data is appropriate for publication.

[0348] The addition of an emotion engine allows the system to analyze the user's facial expressions and voice to recognize their emotions in real time. Emotions are categorized as positive, negative, and neutral, and the server optimizes the information it provides based on the user's emotional state. For example, if a user is showing negative emotions, the system prioritizes providing image data with generally positive content. It also filters content based on specific emotional states to prevent the display of information that may be unpleasant to the user.

[0349] Finally, the server sends the filtered data to the user's terminal. The terminal analyzes the received data and can integrate and display historical image data and current map data on its interface. The user can simultaneously view historical and current information for a specified location.

[0350] Specific example

[0351] For example, consider a scenario where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. Meanwhile, the emotion engine recognizes "positive emotions" from the user's facial expressions and voice. This information is sent to the server. The server searches its database for entries containing historical image data related to the specified location in Shinjuku Ward, retrieving, for example, "a photograph of a residential area in Shinjuku Ward in the 1930s" or "positive episodes from the reconstruction period after the Great Kanto Earthquake." At the same time, the emotion engine optimizes the data to provide overall positive content, taking into account the user's positive emotions. The server filters this data, excluding sensitive information, and then sends the organized data to the user's device. The device receives the data and displays the historical images and their descriptions on the screen along with the current map. The user can easily compare the past and present while prioritizing the viewing of positive information.

[0352] Thus, the present invention is a system that efficiently links past image data and current map data, and further utilizes an emotion engine to provide optimal information tailored to the user's emotional state.

[0353] Example of a prompt

[0354] The following are examples of prompt statements to input into the generating AI model.

[0355] Example 1:

[0356] Search for image data of "Shinjuku Ward, 1930s residential area" and provide information that includes positive episodes. The user's emotional state is positive.

[0357] Example 2:

[0358] Display image data related to the "episode from the reconstruction period after the Great Kanto Earthquake" specified by the user. The emotion engine will analyze the user's emotional state, and if it indicates positive emotions, prioritize providing data with positive content.

[0359] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0360] Step 1:

[0361] Image data collection:

[0362] The server executes a script to automatically retrieve image data from a publicly available database on the internet. Specifically, it uses the Python requests library to send an HTTP request from a specific URL and download the image data. The input for this step is the URL of the public database, and the output is the downloaded image file.

[0363] Specific examples of operation:

[0364] The server accesses https: / / example.com / api / images and downloads the image data.

[0365] Step 2:

[0366] Image data analysis:

[0367] The server analyzes the collected image data using AI image recognition technology (e.g., OpenCV models). The input is the downloaded image data, and the output is metadata about the date, location, and event of the photograph. The AI ​​extracts this metadata from text information and digital prints within the image.

[0368] Specific examples of operation:

[0369] The server uses OpenCV and pytesseract to analyze image data and extract text information.

[0370] Step 3:

[0371] Adding location information:

[0372] The server uses the extracted metadata to add location information (latitude and longitude) to the image data. The input is metadata about the shooting location, and the output is location-attached image data containing latitude and longitude information. In this step, the metadata is sent to a geocoding API to obtain the latitude and longitude.

[0373] Specific examples of operation:

[0374] The server sends location information for the filming location in Shinjuku Ward to a geocoding API to obtain the latitude and longitude.

[0375] Step 4:

[0376] Saving to the database:

[0377] The server stores location-tagged image data in a specialized database. The input is location-tagged image data, and the output is the data entries stored in the database. For storage efficiency, the server stores the image data itself, its links, metadata, location information, etc.

[0378] Specific examples of operation:

[0379] The server connects to a MySQL database and stores image data, including location information and metadata.

[0380] Step 5:

[0381] Receiving user requests:

[0382] Users operating the terminal access the system through a dedicated application or website, specify the location they want to view on a map, and send a request. The input is the location specified by the user, and the output is a search request related to that location.

[0383] Specific examples of operation:

[0384] The user launches the application, clicks on a location in Shinjuku Ward on the map, and sends a request.

[0385] Step 6:

[0386] Searching and filtering data:

[0387] The server searches its database for historical image data corresponding to a specified location based on the received user request. The input is the user's request, and the output is the associated image dataset. Simultaneously, sensitive information is filtered.

[0388] Specific examples of operation:

[0389] The server searches the database for entries related to Shinjuku Ward, filters out sensitive information, and retrieves the results.

[0390] Step 7:

[0391] Emotion recognition by an emotion engine:

[0392] The user's facial expressions and voice are collected through the device's camera and microphone, and the server's emotion engine analyzes them in real time. The input is the user's facial expressions and voice data, and the output is categorized as positive, negative, or neutral emotions.

[0393] Specific examples of operation:

[0394] The server uses EmotionEngine to analyze the user's real-time data and recognizes their emotional state as positive.

[0395] Step 8:

[0396] Optimizing the information provided:

[0397] The server optimizes the information it provides based on the user's emotional state. The input consists of emotion recognition results and historical image data, while the output is an optimized image dataset. For example, if a negative emotion is indicated, it prioritizes image data with positive content.

[0398] Specific examples of operation:

[0399] The server considers the emotional state to be positive and filters out data that includes positive episodes.

[0400] Step 9:

[0401] Sending and displaying data:

[0402] The server sends optimized data to the user's terminal. The input is optimized image data, and the output is data transmission to the terminal. The terminal analyzes the received data and displays it on the interface, integrating historical image data and current map data. This allows the user to view historical and current information for a specified location simultaneously.

[0403] Specific examples of operation:

[0404] The server sends optimized image data to the user's terminal, which then displays it on the screen along with the current map data.

[0405] (Application Example 2)

[0406] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0407] This invention relates to a system that collects and organizes past image data and searches and displays it based on a location specified by the user. However, conventional systems provide uniform information without considering the user's emotional state, resulting in the inability to provide information tailored to the individual user's emotions and needs. As a result, the user experience may be compromised. This invention aims to solve this problem and realize the provision of optimal information according to the user's emotional state.

[0408] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0409] In this invention, the server includes means for collecting past image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving a location specification from a user, means for searching the database based on the location specification, means for providing past image data as a search result, means for integrating and displaying the past image data and current map data, and means for analyzing the user's emotions and optimizing the image data provided based on those emotions. This makes it possible to provide information that takes into account the user's emotional state.

[0410] "Past image data" refers to digital data of photographs and videos taken in the past.

[0411] "Metadata" refers to information associated with image data, including details about the date and location of the photograph, and events that occurred.

[0412] "Location information" refers to data that indicates a geographical location, and is usually represented by latitude and longitude.

[0413] A "database" is a system designed to efficiently store and retrieve large amounts of data.

[0414] "Location specification" refers to the operation in which a user selects or specifies a particular geographical location.

[0415] "Map data" refers to digital data containing geographical information, showing current topography and the locations of buildings.

[0416] "Emotion analysis" is a technology that identifies emotions from a user's facial expressions and voice, and analyzes their emotional state.

[0417] "Searching" is the process of exploring a database based on specified conditions and extracting relevant data.

[0418] "Integrated display" is the operation of displaying multiple data points together on a single screen.

[0419] "Optimization" means adjusting or improving something to best suit a specific purpose.

[0420] This invention combines a system that collects historical image data, organizes it based on metadata, stores it in a database with location information, searches it based on the user's specified location, and displays the integrated historical image data and current map data with an emotion engine that recognizes the user's emotions. This enables the provision of optimal information according to the user's emotional state.

[0421] The system's core is the server. The server handles data processing, storage, and responding to user requests. It also incorporates an emotion recognition engine to analyze user emotions. Specifically, the server includes the following:

[0422] 1. Collection of image data

[0423] The server executes a script to automatically retrieve image data from publicly available databases on the internet. This temporarily stores past image data in storage.

[0424] 2. Extraction of metadata

[0425] The server analyzes the collected image data using AI image recognition technology (specifically, TENSORFLOW® and Keras can be used). The AI ​​extracts metadata from the images, including the date, location, and events that occurred. Based on this metadata, the server adds location information (latitude and longitude) to each image.

[0426] 3. Storing and retrieving data in the database

[0427] The location-tagged image data is stored in a specialized database (MongoDB or PostgreSQL are examples). This database is indexed to enable efficient searching later. Based on the location specified by the user, the server searches the database and retrieves historical image data corresponding to the specified location.

[0428] 4. Emotion Recognition and Optimization

[0429] An emotion recognition engine (specifically, Microsoft® Azure®'s Emotion API or Google Cloud AI's Vision AI can be used) analyzes the user's facial expressions and voice to recognize the user's emotions in real time. Emotions are categorized as positive, negative, and neutral. The server optimizes the information it provides based on the user's emotional state. For example, it prioritizes providing positive image data to a user exhibiting negative emotions.

[0430] 5. Integrated display

[0431] The server sends filtered data to the user's device (smart glasses, smartphone, etc.). The device analyzes the received data and displays it by integrating historical image data and current map data on its interface. For example, map data can be displayed using the Google Maps API or Apple Maps.

[0432] Specific example

[0433] For example, consider a scenario where a user specifies a tourist spot in Shinjuku Ward. The user launches a dedicated application and selects a specific location in Shinjuku Ward on a map. Simultaneously, the user's facial expressions and voice are analyzed via camera and microphone, and an emotion engine recognizes "positive emotions." This information is then sent to the server.

[0434] The server searches its database for historical image data related to a specified location in Shinjuku Ward (e.g., photos of Shinjuku Ward from the 1930s or positive anecdotes) and provides it. The sentiment engine prioritizes selecting data with an overall positive content, taking into account that the user has expressed positive emotions.

[0435] The server filters this data, excluding sensitive information, and then sends the organized data to the user's device. The user's device receives the data and displays past images and their descriptions on the screen along with the current map. In this way, the user can compare the past and present while prioritizing the viewing of positive information.

[0436] Example of a prompt

[0437] "Integrate historical image data with current map data and provide information that matches the user's positive emotions. For example, if a user is moved, display historically positive photos or stories about that place."

[0438] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0439] Step 1:

[0440] Users access the system through smart glasses or a smartphone app and specify a particular geographical location. This input includes information about the geographical location specified by the user (latitude and longitude).

[0441] Specific operation: The user launches a map application and taps or clicks on a specific location in Shinjuku Ward on the map to specify it.

[0442] Step 2:

[0443] The server receives location information sent by the user and searches the database for historical image data associated with that location. The input here is the geographical location information specified by the user, and the output is the corresponding historical image data.

[0444] Specific operation: The server generates a database query to search for image data corresponding to the specified latitude and longitude. MongoDB or PostgreSQL can be used as the database.

[0445] Step 3:

[0446] The server analyzes the collected image data using AI image recognition technology and extracts metadata (date of capture, location, and event). The input here is historical image data, and the output is the associated metadata.

[0447] Specific operation: As AI image recognition technology, TensorFlow and Keras are used to extract necessary information from image data.

[0448] Step 4:

[0449] The server analyzes the user's emotions using data acquired from the user's smart glasses or smartphone camera / microphone. The input is the user's facial expression images and audio data, and the output is identified emotion information.

[0450] Specific operation: An emotion recognition engine (e.g., Microsoft Azure's Emotion API) is used to analyze the user's facial expressions and voice to determine the category of emotion (positive, negative, neutral).

[0451] Step 5:

[0452] The server selects the most suitable image data based on the analyzed sentiment information. The input here is image data with sentiment information and metadata, and the output is historical image data optimized for the user's sentiment.

[0453] Specific operation: The server refers to emotional information and, for example, prioritizes selecting bright and hopeful images for users who exhibit positive emotions.

[0454] Step 6:

[0455] The server sends image data optimized according to the user's emotions to the user's device. Here, the input is the optimized image data, and the output is the content displayed on the user's device.

[0456] Specific operation: The server organizes the selected image data, bundles it into a format such as JSON, and sends it to the user's smart glasses or smartphone application.

[0457] Step 7:

[0458] The terminal integrates and displays historical image data and current map data based on the received data. The input consists of image data and map data sent from the server, and the output is an integrated display screen that the user can view.

[0459] Specific operation: The device will use the Google Maps API or Apple Maps to build an interface that overlays current map data with historical image data.

[0460] Through the above processing steps, users can simultaneously compare the past and present of their location while receiving information optimized for their emotions at that time.

[0461] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0462] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0463] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0464] [Second Embodiment]

[0465] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0466] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0467] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0468] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0469] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0470] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0471] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0472] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0473] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0474] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0475] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0476] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0477] The present invention is a system that collects historical image data, organizes it based on metadata, adds location information and stores it in a database, searches based on the user's specified location, and displays the integrated historical image data and current map data. The following describes in detail how the present invention is implemented.

[0478] System Configuration

[0479] 1. Server: This server is central to the present invention, performing data processing and storage, and responding to user requests.

[0480] 2. Device: A device (such as a smartphone, tablet, or PC) that a user uses to access and operate the system through an application or website.

[0481] 3. User: A person who specifies a location, views the results, and performs operations through a dedicated interface.

[0482] System operation

[0483] First, historical image data is collected. The server executes a script that automatically retrieves image data from publicly available databases on the internet. The collected image data is temporarily stored in storage.

[0484] Next, the server analyzes the collected image data using AI image recognition technology. The AI ​​extracts metadata from the images, including the date, location, and events that occurred. Based on this metadata, the server adds location information (latitude and longitude) to each image.

[0485] The image data with added location information is stored in a specialized database. This database is indexed later to enable efficient searching.

[0486] Users access the system through a dedicated application or website. Users specify a location they want to view on a map and submit a request. The server receives this request and searches its database for historical image data corresponding to the specified location.

[0487] The server organizes the search results and sends them to the user's terminal. The terminal integrates the received historical image data and metadata with current map data and displays it on the screen. This allows the user to compare and verify past and present conditions.

[0488] Specific example

[0489] For example, consider a case where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. This information is sent to the server by the user's action. The server searches its database for entries containing historical image data related to the specified location in Shinjuku Ward, and retrieves relevant data such as "a photograph of a residential area in Shinjuku Ward in the 1930s" or "a depiction of the area during the reconstruction period after the Great Kanto Earthquake."

[0490] The server filters this data, removing sensitive information (such as detailed addresses or images of individuals' faces), and then sends the organized data to the user's device. The device receives the data and displays past images and their descriptions on the screen along with a current map. The user can easily compare the past and present.

[0491] Thus, the present invention is a system that efficiently links historical image data and current map data to provide useful information to users.

[0492] The following describes the processing flow.

[0493] Step 1:

[0494] Server: Executes scripts to access publicly available databases and libraries on the internet and collect historical image data. The collected image data is stored in temporary storage.

[0495] Step 2:

[0496] Server: The server initiates analysis of the collected image data using AI image recognition technology. The AI ​​model extracts metadata from the images, including the date, location, and events that occurred.

[0497] Step 3:

[0498] Server: Based on the extracted metadata, it adds location information such as latitude and longitude to each image. This information is used to set location attributes for the image data.

[0499] Step 4:

[0500] Server: Stores location-tagged image data in a database. An index is created during storage to enable efficient searching later.

[0501] Step 5:

[0502] User: Access the dedicated application or website and log in. The user uses the map interface to specify the location for which they want to view past information by clicking or tapping.

[0503] Step 6:

[0504] Terminal: Sends the user's location information as a request to the server. The request includes the latitude and longitude information of the specified location.

[0505] Step 7:

[0506] Server: Based on the received location request, it searches the database. It retrieves historical image data and its metadata associated with the specified latitude and longitude.

[0507] Step 8:

[0508] Server: Organizes search results and prepares image datasets related to the specified locations. Filters out sensitive information to ensure it is appropriate for publication.

[0509] Step 9:

[0510] Server: Sends filtered data to the user's terminal. The data is sent in a structured format such as JSON.

[0511] Step 10:

[0512] Terminal: Analyzes received data and integrates historical image data and current map data on the interface for display. Users can simultaneously view historical and current information for a specified location.

[0513] Step 11:

[0514] User: Based on the displayed information, users can view more detailed information as needed or based on their interests. Users can also specify other locations and check past information using a similar procedure.

[0515] This series of processes allows users to view historical image data and current map data in an integrated manner, making it easy to understand the historical background and past conditions of a location.

[0516] (Example 1)

[0517] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0518] Conventional technologies had problems with efficiency and accuracy in systems that properly collected, analyzed, and stored historical image data and linked it with location information for searching. Furthermore, the process of integrating historical image data corresponding to a user-specified region with current map data presented challenges in filtering search results and efficiently managing the data.

[0519] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0520] In this invention, the server includes means for collecting historical image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving a location specification from a user, means for searching the database based on the location specification, means for providing historical image data as search results, means for integrating and displaying the historical image data and current map data, means for filtering the search results, and means for performing indexing. This makes it possible to efficiently and accurately collect and store historical image data and provide appropriate search results in response to user requests.

[0521] "Image data" refers to digital image information that records past events and places.

[0522] "Metadata" refers to information associated with the image data itself, including data such as the date and time of shooting, the location of shooting, and details of the event.

[0523] "Location information" refers to digital data of latitude and longitude used to indicate a specific location.

[0524] A "database" is an information management system for efficiently storing, managing, and retrieving collected image data and metadata.

[0525] "Location specification" refers to the geographical location that a user specifies when they wish to obtain information about a particular place.

[0526] "Search results" refer to information extracted from a database of past image data related to the location specified by the user.

[0527] "Filtering" is the process of removing unnecessary data or sensitive information from search results.

[0528] "Indexing" is the process of organizing and indexing data stored in a database so that it can be searched efficiently.

[0529] A "user" is an individual or group that performs operations to obtain information using a system.

[0530] A "server" is a central management system that collects, analyzes, stores, searches, and provides results to users for image data.

[0531] A "terminal" is a device (such as a smartphone, tablet, or PC) that a user uses to access and operate a system.

[0532] The present invention is a system that collects historical image data, organizes it based on metadata, adds location information and stores it in a database, searches based on the user's specified location, and displays the integrated historical image data and current map data. The following describes in detail how the present invention is implemented.

[0533] First, historical image data is collected. The server executes a script that automatically retrieves image data from publicly available databases on the internet. For example, the "Public Image Database API" or the "Image Search API" can be used. The collected image data is temporarily stored in storage (for example, Amazon S3).

[0534] Next, the server analyzes the collected image data using AI image recognition technology (e.g., Google Cloud Vision API). Here, metadata such as the date, location, and event of the image are extracted from the image data. Based on this metadata, the server adds location information (latitude and longitude) to each image. Specifically, location information is obtained using the Google Maps Geocoding API.

[0535] The added location-tagged image data is stored in a database (e.g., PostgreSQL). This database is indexed using Elasticsearch to enable efficient searching later on.

[0536] Users access the system through a dedicated application or website. Users specify a location they want to view on a map and submit a request. This request is received by the server, which searches its database for historical image data corresponding to the specified location. The server then organizes the search results, excluding sensitive information (such as detailed addresses or images of individuals' faces) before sending them to the user's device.

[0537] The device integrates received historical image data and metadata with current map data (e.g., Google Maps API) and displays it on the screen. This allows the user to compare and verify past and present conditions.

[0538] Specific example

[0539] For example, consider a case where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. This information is sent to the server as a result of the user's actions. The server searches its database for entries containing historical image data related to the specified location in Shinjuku Ward and retrieves relevant data such as "a photograph of a residential area in Shinjuku Ward in the 1930s" or "a picture of the area during the disaster recovery period."

[0540] The server filters and organizes this data, then sends the organized data to the user's terminal. The terminal receives the data and displays past images and their descriptions on the screen along with the current map. The user can easily compare the past and the present.

[0541] Examples of prompts to input into a generative AI model

[0542] Example of a prompt:

[0543] Please provide a detailed explanation of how a system works to integrate "photographs of residential areas in Shinjuku Ward from the 1930s" with "a current map of Shinjuku Ward."

[0544] Please describe the specific steps involved in each stage of the process.

[0545] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0546] Step 1:

[0547] The server automatically retrieves image data from publicly available databases on the internet.

[0548] Input: API endpoint for the public image database.

[0549] Data processing / calculation: Send an API request and temporarily store the retrieved image data.

[0550] Output: Temporarily saved image data.

[0551] Specific operation: A Python script is executed to send requests to a public database API and save the retrieved image data to storage such as Amazon S3.

[0552] Step 2:

[0553] The server analyzes the collected image data using AI image recognition technology.

[0554] Input: Temporarily saved image data.

[0555] Data processing / calculation: Send images to an AI image recognition service and extract metadata such as the date and location of the photo, and information about the event.

[0556] Output: Image data with metadata attached.

[0557] Specific operation: Use the Google Cloud Vision API to analyze image data and retrieve metadata including the date and time and location of the image.

[0558] Step 3:

[0559] The server adds latitude and longitude location information to the image based on the metadata.

[0560] Input: Image data with metadata attached.

[0561] Data processing / calculation: Use the Geocoding API to convert text information (address / place name) into latitude and longitude, and add location information to image data.

[0562] Output: Image data with latitude and longitude information added.

[0563] Specific operation: Use the Google Maps Geocoding API to parse the address information in the metadata and obtain the appropriate latitude and longitude location information.

[0564] Step 4:

[0565] The server stores location-tagged image data in a database.

[0566] Input: Image data with latitude and longitude information attached.

[0567] Data processing / calculation: Insert image data into the database and create indexes to enable efficient searching.

[0568] Output: Image data with location information stored in the database.

[0569] Specific operation: Store image data in a PostgreSQL database and perform indexing using Elasticsearch.

[0570] Step 5:

[0571] Users access the system through a dedicated application or website, specify the location they want to view on a map, and submit a request.

[0572] Input: Geographic location information specified by the user.

[0573] Data processing / calculation: Send a request to the server and query information from the specified location.

[0574] Output: Request received by the server.

[0575] Specific operation: The user uses an application or website to click on a location they want to see on a map and sends that information to the server.

[0576] Step 6:

[0577] The server receives a user request and searches the database for historical image data corresponding to the specified location.

[0578] Input: Location request received from the user.

[0579] Data processing / calculation: Search the database for image data corresponding to the queried location and perform filtering.

[0580] Output: Filtered search results.

[0581] Specific operation: Execute Elasticsearch queries to search the database for appropriate image data and exclude sensitive information.

[0582] Step 7:

[0583] The server organizes the search results and sends them to the user's device.

[0584] Input: Filtered search results.

[0585] Data processing / calculation: Organize search results and convert them into a format that can be sent to the user.

[0586] Output: The organized data sent to the user's terminal.

[0587] Specific operation: Convert the search results to JSON format and send them to the user's terminal as an HTTP response.

[0588] Step 8:

[0589] The terminal integrates the received historical image data and metadata with current map data and displays it on the screen.

[0590] Input: Data sent from the server.

[0591] Data Processing / Calculation: Process the received data into a format that can be displayed by integrating it with the map API.

[0592] Output: Integrated data displayed on the user's screen.

[0593] Specific operation: Use the Google Maps API to integrate historical image data with current map data and display it on the user's device.

[0594] (Application Example 1)

[0595] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0596] Conventional technologies had limited means of collecting and effectively providing historical image data to users, making it difficult to compare past and present situations. Furthermore, there was no system that could provide users with relevant contemporary product information based on their interests and enable purchases. Therefore, there is a need to improve the convenience for users when purchasing related products using historical information.

[0597] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0598] In this invention, the server includes means for collecting past image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving a location specification from a user, means for searching the database based on the location specification, means for providing past image data as a search result, means for integrating and displaying the past image data and current map data, means for displaying past and current image data and providing explanatory text, and means for providing related product information based on the user's interests and enabling purchase. This makes it possible for the user to easily compare past and present situations and effectively purchase related products.

[0599] "Past image data" refers to image data taken at a specific time, representing past events or scenes.

[0600] "Metadata" refers to attribute information associated with image data, including information such as the date and time of shooting, location, and events.

[0601] "Location information" refers to geographical information such as latitude and longitude that indicates the location where the image data was taken.

[0602] A "database" is a system that systematically stores and manages digital data, enabling efficient searching.

[0603] "Location specification" is the operation of specifying a particular geographical location that the user is interested in.

[0604] "Searching" is the operation of finding data within a database based on specific criteria.

[0605] "Current map data" refers to digital map data that displays the latest geographical information.

[0606] "Integrated display" means displaying different data together on a single screen or in a single format.

[0607] "Displaying past and present image data" means displaying past image data and current map data on the screen simultaneously.

[0608] "Description" refers to additional information or explanatory text related to image data.

[0609] "Related product information" refers to product data provided based on the user's interests.

[0610] "Means of enabling purchase" refers to a function that allows users to directly purchase products they are interested in.

[0611] This invention is a system that collects historical image data, organizes it based on metadata, adds location information, stores it in a database, searches based on the user's specified location, and displays the integrated historical image data and current map data. Furthermore, it includes a function to provide relevant product information based on the user's interests and enable purchases.

[0612] The server is at the heart of the system and performs the following roles:

[0613] 1. Server

[0614] The server collects historical image data, extracts metadata, adds location information, and stores it in a database. Furthermore, it receives location information from the user, searches the database based on that information, and provides historical image data as search results. The server integrates current map data with historical image data and displays it, providing the user with relevant product information.

[0615] 2. Terminal

[0616] A terminal is a device that users access and operate through a dedicated application. This includes smartphones, tablets, and PCs. The terminal displays historical image data and current map data received from the server, and displays explanatory text and related product information according to user requests.

[0617] 3. User

[0618] Users specify their location through a dedicated interface and view and manipulate data provided by the system. They can compare past and present situations, obtain relevant product information, and purchase items of interest directly.

[0619] Specific example

[0620] The following is a specific example of the process.

[0621] The user launches the "Time Travel Shopping App" on their smartphone and specifies a particular location on the map, for example, "Shinjuku Ward." They also enter a time period for which they want to view past images, such as "1930-1940." This information is sent from the device to the server. The server searches its database based on the specified latitude, longitude, and date range and retrieves the corresponding past image data.

[0622] The acquired image data includes metadata such as the date and location of the photograph, and this information is transmitted to the device along with the image data. The device integrates the past image data with current map data and displays the past and present images simultaneously. A descriptive text related to the image is also displayed.

[0623] Furthermore, the server provides relevant product information based on the user's interests. For example, if an image of a shopping street from the 1930s is displayed, products that were sold at that time and related modern items will be introduced. Users can purchase products they are interested in directly within the app.

[0624] Example of a prompt

[0625] You are an assistant that searches historical image data and integrates it with current map data for display. When given a location name and date range, search for historical image data of that location and display it compared to current map data. Process the following requests:

[0626] Location: Shinjuku Ward

[0627] Start date: 1930-01-01

[0628] End date: 1940-12-31

[0629] This invention allows users to easily access historical information and compare it with the present to gain a deeper historical understanding. Furthermore, by providing related product information, users can proceed directly to the purchase process, significantly improving convenience.

[0630] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0631] Step 1:

[0632] The user specifies a location and time period on a map.

[0633] The user launches the "Time Travel Shopping App" on their smartphone, specifies a location of interest on the map, and enters the period (start and end dates) for which they want to view past images. The entered information is then sent from the device to the server.

[0634] input

[0635] Information about the specified location (e.g., Shinjuku Ward)

[0636] Specified period (start date, end date)

[0637] output

[0638] Request data to the server (location information and time period information)

[0639] Specific actions

[0640] The terminal generates an API request that sends the specified location and time period to the server via the user interface.

[0641] Step 2:

[0642] The server searches for past image data.

[0643] The server searches the database based on the received request data (location information and time period information) to retrieve relevant historical image data. The server uses indexing to perform efficient searches based on location information and time period.

[0644] input

[0645] Request data (location information and time period information)

[0646] output

[0647] Search results (corresponding past image data)

[0648] Specific actions

[0649] The server executes database queries based on the specified latitude, longitude, and date ranges and collects the corresponding image data.

[0650] Step 3:

[0651] The server analyzes the image data and extracts the metadata.

[0652] The server analyzes the acquired image data using AI image recognition technology and extracts metadata for each image, including the date and location of the image, as well as information about related events.

[0653] input

[0654] Search results (past image data)

[0655] output

[0656] Image data with metadata

[0657] Specific actions

[0658] The server uses an AI model (e.g., an image recognition model) to analyze image data and automatically extract metadata such as the date and location of the photograph, and information about the event.

[0659] Step 4:

[0660] The server adds location information and stores it in the database.

[0661] The server adds location information (latitude and longitude) to each image data based on the extracted metadata and stores it in the database as image data with location information.

[0662] input

[0663] Image data with metadata

[0664] output

[0665] Image data with location information

[0666] Specific actions

[0667] The server calculates the latitude and longitude information corresponding to each image based on the metadata, adds location information, and stores it in the database.

[0668] Step 5:

[0669] The server integrates current map data and historical image data and transmits it.

[0670] The server integrates location-based image data with current map data, converts it into a data format that allows the user to compare past images with the current map, and sends it to the terminal.

[0671] input

[0672] Image data with location information

[0673] Current map data

[0674] output

[0675] Integrated data (data from past images and current maps)

[0676] Specific actions

[0677] The server overlays historical images onto map data based on location information and converts them into a format that allows for comparison and display of both.

[0678] Step 6:

[0679] The device displays integrated data and provides explanatory text.

[0680] The device displays the received integrated data and compares past and present images. It also displays explanatory text related to the images.

[0681] input

[0682] Integrated data (data from past images and current maps)

[0683] output

[0684] Images and descriptions displayed on the user interface

[0685] Specific actions

[0686] The device displays integrated data in a user-friendly format and provides a convenient interface for comparing past and present situations.

[0687] Step 7:

[0688] The server provides relevant product information based on the user's interests.

[0689] The server generates relevant product information based on past image data, descriptions, and user operation data, and transmits this information to the terminal.

[0690] input

[0691] User operation data

[0692] Past image data

[0693] Description

[0694] output

[0695] Related product information

[0696] Specific actions

[0697] The server uses a generative AI model to analyze user interests and generate data to select and provide corresponding product information.

[0698] Step 8:

[0699] The device displays related product information and enables the purchase process.

[0700] The device displays the received related product information, allowing the user to purchase items they are interested in directly.

[0701] input

[0702] Related product information

[0703] output

[0704] Product information and purchase options displayed on the user interface

[0705] Specific actions

[0706] The terminal displays related product information and provides an interface that allows users to proceed directly to the purchase process by clicking on a product.

[0707] The above outlines the specific processing steps. This processing flow allows users to easily compare past and current situations and gain an effective experience in purchasing related products.

[0708] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0709] This invention combines a system that collects historical image data, organizes it based on metadata, stores it in a database with location information, searches it based on the user's specified location, and displays the integrated historical image data and current map data with an emotion engine that recognizes the user's emotions. This enables the provision of optimal information according to the user's emotional state.

[0710] System Configuration

[0711] 1. Server: This server is central to the present invention, performing data processing and storage, and responding to user requests. It also incorporates an emotion engine to analyze user emotions.

[0712] 2. Device: A device (such as a smartphone, tablet, or PC) that a user uses to access and operate the system through an application or website.

[0713] 3. User: A person who specifies a location, views the results, and performs operations through a dedicated interface.

[0714] System operation

[0715] First, historical image data is collected. The server executes a script that automatically retrieves image data from publicly available databases on the internet. The collected image data is temporarily stored in storage.

[0716] Next, the server analyzes the collected image data using AI image recognition technology. The AI ​​extracts metadata from the images, including the date, location, and events that occurred. Based on this metadata, the server adds location information (latitude and longitude) to each image.

[0717] The image data with added location information is stored in a specialized database. This database is indexed later to enable efficient searching.

[0718] Users access the system through a dedicated application or website. Users specify a location they want to view on a map and submit a request. The server receives this request and searches its database for historical image data corresponding to the specified location.

[0719] The server organizes the search results and prepares image datasets related to the specified locations. It filters out any sensitive information to ensure that the images are suitable for publication.

[0720] This is where the emotion engine comes in. The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions in real time. Emotions are categorized into positive, negative, and neutral.

[0721] The server optimizes the information it provides based on the user's emotional state. For example, if a user is showing negative emotions, it prioritizes providing image data with generally positive content. It also performs filtering corresponding to specific emotional states to prevent the display of information that may be offensive to the user.

[0722] Finally, the server sends the filtered data to the user's terminal. The terminal analyzes the received data and displays the historical image data and current map data integrated on the interface. The user can simultaneously view historical and current information for a specified location.

[0723] Specific example

[0724] For example, consider a scenario where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. Meanwhile, the emotion engine recognizes "positive emotions" from the user's facial expressions and voice. This information is sent to the server.

[0725] The server searches the database for entries containing historical image data related to a specified location in Shinjuku Ward, retrieving entries such as "a photograph of a residential area in Shinjuku Ward in the 1930s" or "positive anecdotes from the reconstruction period after the Great Kanto Earthquake." Simultaneously, the sentiment engine optimizes the data to provide an overall positive content, taking into account the user's positive emotions.

[0726] The server filters this data, excluding sensitive information, and then sends the organized data to the user's device. The device receives the data and displays past images and their descriptions on the screen along with a current map. Users can easily compare the past and present while prioritizing the viewing of positive information.

[0727] Thus, the present invention is a system that efficiently links past image data and current map data, and further utilizes an emotion engine to provide optimal information tailored to the user's emotional state.

[0728] The following describes the processing flow.

[0729] Step 1:

[0730] Server: Executes scripts to access publicly available databases and libraries on the internet and collect historical image data. The collected image data is stored in temporary storage.

[0731] Step 2:

[0732] Server: The server initiates analysis of the collected image data using AI image recognition technology. The AI ​​model extracts metadata from the images, including the date, location, and events that occurred.

[0733] Step 3:

[0734] Server: Based on the extracted metadata, it adds location information such as latitude and longitude to each image. This information is used to set location attributes for the image data.

[0735] Step 4:

[0736] Server: Stores location-tagged image data in a database. An index is created during storage to enable efficient searching later.

[0737] Step 5:

[0738] User: Access the dedicated application or website and log in. The user uses the map interface to specify the location for which they want to view past information by clicking or tapping.

[0739] Step 6:

[0740] Terminal: Sends the user's location information as a request to the server. The request includes the latitude and longitude information of the specified location.

[0741] Step 7:

[0742] Server: Based on the received location request, it searches the database. It retrieves historical image data and its metadata associated with the specified latitude and longitude.

[0743] Step 8:

[0744] Server: Organizes search results and prepares image datasets related to the specified locations. Filters out sensitive information to ensure it is appropriate for publication.

[0745] Step 9:

[0746] Server: Sends filtered data to the user's terminal. The data is sent in a structured format such as JSON.

[0747] Step 10:

[0748] Terminal: Analyzes received data and integrates historical image data and current map data on the interface for display. Users can simultaneously view historical and current information for a specified location.

[0749] Step 11:

[0750] User: Uses the device's camera and microphone to express emotions through facial expressions and voice. The emotion engine acquires this data and analyzes emotions in real time.

[0751] Step 12:

[0752] Server: The emotion engine classifies the user's emotions as positive, negative, or neutral, and provides optimized information based on that emotional state.

[0753] Step 13:

[0754] Server: Filters relevant metadata based on emotional state. For example, if a user is expressing negative emotions, it prioritizes providing image data with positive content.

[0755] Step 14:

[0756] Server: Retransmits the information set optimized by the emotion engine to the user's terminal.

[0757] Step 15:

[0758] Device: Redisplays past images and their descriptions that match the user's emotional state, along with the current map. Users can enjoy viewing information in an emotionally sensitive way.

[0759] This series of steps allows users to view historical image data and current map data in an integrated manner, and then receive optimized information from the emotion engine.

[0760] (Example 2)

[0761] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0762] Conventional systems that collect historical image data, add location information, store it in a database, and search based on the user's specified location do not consider the user's emotional state, and the information provided may not always be optimal. Furthermore, if sensitive information is not filtered, inappropriate information may be provided. To solve this problem, a system is needed that provides optimal information tailored to the user's emotional state.

[0763] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting past image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the image data with location information in a database, means for receiving a location specification from the user, means for searching the database based on the location specification, means for providing past image data as a search result, means for integrating and displaying past image data and current map data, and an emotion engine that recognizes the user's emotions, analyzes the user's emotions using the emotion engine, and optimizes the information to be provided based on the analysis results. This makes it possible to provide optimal information according to the user's emotional state.

[0764] "Past image data" refers to photographs and image files taken in the past, including data such as the date, time, and location of the shooting.

[0765] "Metadata" refers to supplementary information related to image data, including data such as the date and time of shooting, location, and details of the event.

[0766] "Location information" refers to longitude and latitude data used to indicate a specific location.

[0767] A "database" refers to a specialized data storage system for efficiently storing, searching, and managing image data, its metadata, and location information.

[0768] A "user" refers to a person who specifies a location or performs other operations through the system.

[0769] "Location specification" refers to the user selecting a location they want to see on a map or specifying it in text.

[0770] An "emotion engine" refers to a system equipped with technology that analyzes a user's emotions in real time from their facial expressions and voice, and classifies that emotional state as positive, negative, neutral, etc.

[0771] "Integration" refers to combining historical image data and current map data and displaying them on a single interface.

[0772] "Optimization" refers to adjusting the content and order of information provided based on the user's emotional state, as analyzed by the emotion engine, to make the user experience as effective as possible.

[0773] "Sensitive information" refers to information that should be restricted from public disclosure based on specific criteria, such as personal information or data deemed socially inappropriate.

[0774] Modes for carrying out the invention

[0775] This invention combines a system that collects historical image data, organizes it based on metadata, stores it in a database with location information, searches it based on the user's specified location, and displays the integrated historical image data and current map data with an emotion engine that recognizes the user's emotions. This enables the provision of optimal information according to the user's emotional state.

[0776] Hardware and software configuration

[0777] 1. Server:

[0778] The server performs the central data processing and storage of this invention, as well as responding to user requests. It also incorporates an emotion engine to analyze user emotions. This requires a high-performance server, utilizing software such as Python, MySQL, and OpenCV.

[0779] 2. Terminal:

[0780] A terminal is a device that a user uses to access and operate a system through an application or website. Typically, smartphones, tablets, and PCs are used.

[0781] 3. User:

[0782] A user is a person who specifies a location, views the results, and performs actions through a dedicated interface.

[0783] Program Processing Description

[0784] First, historical image data is collected. The server executes a script to automatically retrieve image data from publicly available databases on the internet. The collected image data is temporarily stored in the server's storage. Next, the server analyzes the collected image data using AI image recognition technology (e.g., OpenCV models). The AI ​​extracts metadata from the images, such as the date and location of the image, and events, and adds location information (latitude and longitude) based on this metadata. The image data with location information is stored in a specialized database, such as a MySQL database, and indexed to enable efficient searching.

[0785] Users access the system through a dedicated application or website. Users specify a location they wish to view on a map and submit a request. The server receives this request and searches its database for historical image data corresponding to the specified location. The server organizes the search results and prepares an image dataset related to the specified location. During this process, it filters out sensitive information and verifies whether the data is appropriate for publication.

[0786] The addition of an emotion engine allows the system to analyze the user's facial expressions and voice to recognize their emotions in real time. Emotions are categorized as positive, negative, and neutral, and the server optimizes the information it provides based on the user's emotional state. For example, if a user is showing negative emotions, the system prioritizes providing image data with generally positive content. It also filters content based on specific emotional states to prevent the display of information that may be unpleasant to the user.

[0787] Finally, the server sends the filtered data to the user's terminal. The terminal analyzes the received data and can integrate and display historical image data and current map data on its interface. The user can simultaneously view historical and current information for a specified location.

[0788] Specific example

[0789] For example, consider a scenario where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. Meanwhile, the emotion engine recognizes "positive emotions" from the user's facial expressions and voice. This information is sent to the server. The server searches its database for entries containing historical image data related to the specified location in Shinjuku Ward, retrieving, for example, "a photograph of a residential area in Shinjuku Ward in the 1930s" or "positive episodes from the reconstruction period after the Great Kanto Earthquake." At the same time, the emotion engine optimizes the data to provide overall positive content, taking into account the user's positive emotions. The server filters this data, excluding sensitive information, and then sends the organized data to the user's device. The device receives the data and displays the historical images and their descriptions on the screen along with the current map. The user can easily compare the past and present while prioritizing the viewing of positive information.

[0790] Thus, the present invention is a system that efficiently links past image data and current map data, and further utilizes an emotion engine to provide optimal information tailored to the user's emotional state.

[0791] Example of a prompt

[0792] The following are examples of prompt statements to input into the generating AI model.

[0793] Example 1:

[0794] Search for image data of "Shinjuku Ward, 1930s residential area" and provide information that includes positive episodes. The user's emotional state is positive.

[0795] Example 2:

[0796] Display image data related to the "episode from the reconstruction period after the Great Kanto Earthquake" specified by the user. The emotion engine will analyze the user's emotional state, and if it indicates positive emotions, prioritize providing data with positive content.

[0797] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0798] Step 1:

[0799] Image data collection:

[0800] The server executes a script to automatically retrieve image data from a publicly available database on the internet. Specifically, it uses the Python requests library to send an HTTP request from a specific URL and download the image data. The input for this step is the URL of the public database, and the output is the downloaded image file.

[0801] Specific examples of operation:

[0802] The server accesses https: / / example.com / api / images and downloads the image data.

[0803] Step 2:

[0804] Image data analysis:

[0805] The server analyzes the collected image data using AI image recognition technology (e.g., OpenCV models). The input is the downloaded image data, and the output is metadata about the date, location, and event of the photograph. The AI ​​extracts this metadata from text information and digital prints within the image.

[0806] Specific examples of operation:

[0807] The server uses OpenCV and pytesseract to analyze image data and extract text information.

[0808] Step 3:

[0809] Adding location information:

[0810] The server uses the extracted metadata to add location information (latitude and longitude) to the image data. The input is metadata about the shooting location, and the output is location-attached image data containing latitude and longitude information. In this step, the metadata is sent to a geocoding API to obtain the latitude and longitude.

[0811] Specific examples of operation:

[0812] The server sends location information for the filming location in Shinjuku Ward to a geocoding API to obtain the latitude and longitude.

[0813] Step 4:

[0814] Saving to the database:

[0815] The server stores location-tagged image data in a specialized database. The input is location-tagged image data, and the output is the data entries stored in the database. For storage efficiency, the server stores the image data itself, its links, metadata, location information, etc.

[0816] Specific examples of operation:

[0817] The server connects to a MySQL database and stores image data, including location information and metadata.

[0818] Step 5:

[0819] Receiving user requests:

[0820] Users operating the terminal access the system through a dedicated application or website, specify the location they want to view on a map, and send a request. The input is the location specified by the user, and the output is a search request related to that location.

[0821] Specific examples of operation:

[0822] The user launches the application, clicks on a location in Shinjuku Ward on the map, and sends a request.

[0823] Step 6:

[0824] Searching and filtering data:

[0825] The server searches its database for historical image data corresponding to a specified location based on the received user request. The input is the user's request, and the output is the associated image dataset. Simultaneously, sensitive information is filtered.

[0826] Specific examples of operation:

[0827] The server searches the database for entries related to Shinjuku Ward, filters out sensitive information, and retrieves the results.

[0828] Step 7:

[0829] Emotion recognition by an emotion engine:

[0830] The user's facial expressions and voice are collected through the device's camera and microphone, and the server's emotion engine analyzes them in real time. The input is the user's facial expressions and voice data, and the output is categorized as positive, negative, or neutral emotions.

[0831] Specific examples of operation:

[0832] The server uses EmotionEngine to analyze the user's real-time data and recognizes their emotional state as positive.

[0833] Step 8:

[0834] Optimizing the information provided:

[0835] The server optimizes the information it provides based on the user's emotional state. The input consists of emotion recognition results and historical image data, while the output is an optimized image dataset. For example, if a negative emotion is indicated, positive image data is prioritized.

[0836] Specific examples of operation:

[0837] The server considers the emotional state to be positive and filters out data that includes positive episodes.

[0838] Step 9:

[0839] Sending and displaying data:

[0840] The server sends optimized data to the user's terminal. The input is optimized image data, and the output is data transmission to the terminal. The terminal analyzes the received data and displays it on the interface, integrating historical image data and current map data. This allows the user to view historical and current information for a specified location simultaneously.

[0841] Specific examples of operation:

[0842] The server sends optimized image data to the user's terminal, which then displays it on the screen along with the current map data.

[0843] (Application Example 2)

[0844] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0845] This invention relates to a system that collects and organizes past image data and searches and displays it based on a location specified by the user. However, conventional systems provide uniform information without considering the user's emotional state, resulting in the inability to provide information tailored to the individual user's emotions and needs. As a result, the user experience may be compromised. This invention aims to solve this problem and realize the provision of optimal information according to the user's emotional state.

[0846] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0847] In this invention, the server includes means for collecting past image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving a location specification from a user, means for searching the database based on the location specification, means for providing past image data as a search result, means for integrating and displaying the past image data and current map data, and means for analyzing the user's emotions and optimizing the image data provided based on those emotions. This makes it possible to provide information that takes into account the user's emotional state.

[0848] "Past image data" refers to digital data of photographs and videos taken in the past.

[0849] "Metadata" refers to information associated with image data, including details about the date and location of the photograph, and events that occurred.

[0850] "Location information" refers to data that indicates a geographical location, and is usually represented by latitude and longitude.

[0851] A "database" is a system designed to efficiently store and retrieve large amounts of data.

[0852] "Location specification" refers to the operation in which a user selects or specifies a particular geographical location.

[0853] "Map data" refers to digital data containing geographical information, showing current topography and the locations of buildings.

[0854] "Emotion analysis" is a technology that identifies emotions from a user's facial expressions and voice, and analyzes their emotional state.

[0855] "Searching" is the process of exploring a database based on specified conditions and extracting relevant data.

[0856] "Integrated display" is the operation of displaying multiple data points together on a single screen.

[0857] "Optimization" means adjusting or improving something to best suit a specific purpose.

[0858] This invention combines a system that collects historical image data, organizes it based on metadata, stores it in a database with location information, searches it based on the user's specified location, and displays the integrated historical image data and current map data with an emotion engine that recognizes the user's emotions. This enables the provision of optimal information according to the user's emotional state.

[0859] The system's core is the server. The server handles data processing, storage, and responding to user requests. It also incorporates an emotion recognition engine to analyze user emotions. Specifically, the server includes the following:

[0860] 1. Collection of image data

[0861] The server executes a script to automatically retrieve image data from publicly available databases on the internet. This temporarily stores past image data in storage.

[0862] 2. Extraction of metadata

[0863] The server analyzes the collected image data using AI image recognition technology (specifically, TensorFlow and Keras can be used). The AI ​​extracts metadata from the images, including the date, location, and events that occurred. Based on this metadata, the server adds location information (latitude and longitude) to each image.

[0864] 3. Storing and retrieving data in the database

[0865] The location-tagged image data is stored in a specialized database (MongoDB or PostgreSQL are examples). This database is indexed to enable efficient searching later. Based on the location specified by the user, the server searches the database and retrieves historical image data corresponding to the specified location.

[0866] 4. Emotion Recognition and Optimization

[0867] An emotion recognition engine (specifically, Microsoft Azure's Emotion API or Google Cloud AI's Vision AI can be used) analyzes the user's facial expressions and voice to recognize their emotions in real time. Emotions are categorized as positive, negative, and neutral. The server optimizes the information it provides based on the user's emotional state. For example, it prioritizes providing positive image data to a user exhibiting negative emotions.

[0868] 5. Integrated display

[0869] The server sends filtered data to the user's device (smart glasses, smartphone, etc.). The device analyzes the received data and displays it by integrating historical image data and current map data on its interface. For example, map data can be displayed using the Google Maps API or Apple Maps.

[0870] Specific example

[0871] For example, consider a scenario where a user specifies a tourist spot in Shinjuku Ward. The user launches a dedicated application and selects a specific location in Shinjuku Ward on a map. Simultaneously, the user's facial expressions and voice are analyzed via camera and microphone, and an emotion engine recognizes "positive emotions." This information is then sent to the server.

[0872] The server searches its database for historical image data related to a specified location in Shinjuku Ward (e.g., photos of Shinjuku Ward from the 1930s or positive anecdotes) and provides it. The sentiment engine prioritizes selecting data with an overall positive content, taking into account that the user has expressed positive emotions.

[0873] The server filters this data, excluding sensitive information, and then sends the organized data to the user's device. The user's device receives the data and displays past images and their descriptions on the screen along with the current map. In this way, the user can compare the past and present while prioritizing the viewing of positive information.

[0874] Example of a prompt

[0875] "Integrate historical image data with current map data and provide information that matches the user's positive emotions. For example, if a user is moved, display historically positive photos or stories about that place."

[0876] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0877] Step 1:

[0878] Users access the system through smart glasses or a smartphone app and specify a particular geographical location. This input includes information about the geographical location specified by the user (latitude and longitude).

[0879] Specific operation: The user launches a map application and taps or clicks on a specific location in Shinjuku Ward on the map to specify it.

[0880] Step 2:

[0881] The server receives location information sent by the user and searches the database for historical image data associated with that location. The input here is the geographical location information specified by the user, and the output is the corresponding historical image data.

[0882] Specific operation: The server generates a database query to search for image data corresponding to the specified latitude and longitude. MongoDB or PostgreSQL can be used as the database.

[0883] Step 3:

[0884] The server analyzes the collected image data using AI image recognition technology and extracts metadata (date of capture, location, and event). The input here is historical image data, and the output is the associated metadata.

[0885] Specific operation: As AI image recognition technology, TensorFlow and Keras are used to extract necessary information from image data.

[0886] Step 4:

[0887] The server analyzes the user's emotions using data acquired from the user's smart glasses or smartphone camera / microphone. The input is the user's facial expression images and audio data, and the output is identified emotion information.

[0888] Specific operation: An emotion recognition engine (e.g., Microsoft Azure's Emotion API) is used to analyze the user's facial expressions and voice to determine the category of emotion (positive, negative, neutral).

[0889] Step 5:

[0890] The server selects the most suitable image data based on the analyzed sentiment information. The input here is image data with sentiment information and metadata, and the output is historical image data optimized for the user's sentiment.

[0891] Specific operation: The server refers to emotional information and, for example, prioritizes selecting bright and hopeful images for users who exhibit positive emotions.

[0892] Step 6:

[0893] The server sends image data optimized according to the user's emotions to the user's device. Here, the input is the optimized image data, and the output is the content displayed on the user's device.

[0894] Specific operation: The server organizes the selected image data, bundles it into a format such as JSON, and sends it to the user's smart glasses or smartphone application.

[0895] Step 7:

[0896] The terminal integrates and displays historical image data and current map data based on the received data. The input consists of image data and map data sent from the server, and the output is an integrated display screen that the user can view.

[0897] Specific operation: The device will use the Google Maps API or Apple Maps to build an interface that overlays current map data with historical image data.

[0898] Through the above processing steps, users can simultaneously compare the past and present of their location while receiving information optimized for their emotions at that time.

[0899] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0900] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0901] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0902] [Third Embodiment]

[0903] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0904] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0905] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0906] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0907] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0908] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0909] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0910] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0911] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0912] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0913] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0914] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0915] The present invention is a system that collects historical image data, organizes it based on metadata, adds location information and stores it in a database, searches based on the user's specified location, and displays the integrated historical image data and current map data. The following describes in detail how the present invention is implemented.

[0916] System Configuration

[0917] 1. Server: This server is central to the present invention, performing data processing and storage, and responding to user requests.

[0918] 2. Device: A device (such as a smartphone, tablet, or PC) that a user uses to access and operate the system through an application or website.

[0919] 3. User: A person who specifies a location, views the results, and performs operations through a dedicated interface.

[0920] System operation

[0921] First, historical image data is collected. The server executes a script that automatically retrieves image data from publicly available databases on the internet. The collected image data is temporarily stored in storage.

[0922] Next, the server analyzes the collected image data using AI image recognition technology. The AI ​​extracts metadata from the images, including the date, location, and events that occurred. Based on this metadata, the server adds location information (latitude and longitude) to each image.

[0923] The image data with added location information is stored in a specialized database. This database is indexed later to enable efficient searching.

[0924] Users access the system through a dedicated application or website. Users specify a location they want to view on a map and submit a request. The server receives this request and searches its database for historical image data corresponding to the specified location.

[0925] The server organizes the search results and sends them to the user's terminal. The terminal integrates the received historical image data and metadata with current map data and displays it on the screen. This allows the user to compare and verify past and present conditions.

[0926] Specific example

[0927] For example, consider a case where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. This information is sent to the server by the user's action. The server searches its database for entries containing historical image data related to the specified location in Shinjuku Ward, and retrieves relevant data such as "a photograph of a residential area in Shinjuku Ward in the 1930s" or "a depiction of the area during the reconstruction period after the Great Kanto Earthquake."

[0928] The server filters this data, removing sensitive information (such as detailed addresses or images of individuals' faces), and then sends the organized data to the user's device. The device receives the data and displays past images and their descriptions on the screen along with a current map. The user can easily compare the past and present.

[0929] Thus, the present invention is a system that efficiently links historical image data and current map data to provide useful information to users.

[0930] The following describes the processing flow.

[0931] Step 1:

[0932] Server: Executes scripts to access publicly available databases and libraries on the internet and collect historical image data. The collected image data is stored in temporary storage.

[0933] Step 2:

[0934] Server: The server initiates analysis of the collected image data using AI image recognition technology. The AI ​​model extracts metadata from the images, including the date, location, and events that occurred.

[0935] Step 3:

[0936] Server: Based on the extracted metadata, it adds location information such as latitude and longitude to each image. This information is used to set location attributes for the image data.

[0937] Step 4:

[0938] Server: Stores location-tagged image data in a database. An index is created during storage to enable efficient searching later.

[0939] Step 5:

[0940] User: Access the dedicated application or website and log in. The user uses the map interface to specify the location for which they want to view past information by clicking or tapping.

[0941] Step 6:

[0942] Terminal: Sends the user's location information as a request to the server. The request includes the latitude and longitude information of the specified location.

[0943] Step 7:

[0944] Server: Based on the received location request, it searches the database. It retrieves historical image data and its metadata associated with the specified latitude and longitude.

[0945] Step 8:

[0946] Server: Organizes search results and prepares image datasets related to the specified locations. Filters out sensitive information to ensure it is appropriate for publication.

[0947] Step 9:

[0948] Server: Sends filtered data to the user's terminal. The data is sent in a structured format such as JSON.

[0949] Step 10:

[0950] Terminal: Analyzes received data and integrates historical image data and current map data on the interface for display. Users can simultaneously view historical and current information for a specified location.

[0951] Step 11:

[0952] User: Based on the displayed information, users can view more detailed information as needed or based on their interests. Users can also specify other locations and check past information using a similar procedure.

[0953] This series of processes allows users to view historical image data and current map data in an integrated manner, making it easy to understand the historical background and past conditions of a location.

[0954] (Example 1)

[0955] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0956] Conventional technologies had problems with efficiency and accuracy in systems that properly collected, analyzed, and stored historical image data and linked it with location information for searching. Furthermore, the process of integrating historical image data corresponding to a user-specified region with current map data presented challenges in filtering search results and efficiently managing the data.

[0957] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0958] In this invention, the server includes means for collecting historical image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving a location specification from a user, means for searching the database based on the location specification, means for providing historical image data as search results, means for integrating and displaying the historical image data and current map data, means for filtering the search results, and means for performing indexing. This makes it possible to efficiently and accurately collect and store historical image data and provide appropriate search results in response to user requests.

[0959] "Image data" refers to digital image information that records past events and places.

[0960] "Metadata" refers to information associated with the image data itself, including data such as the date and time of shooting, the location of shooting, and details of the event.

[0961] "Location information" refers to digital data of latitude and longitude used to indicate a specific location.

[0962] A "database" is an information management system for efficiently storing, managing, and retrieving collected image data and metadata.

[0963] "Location specification" refers to the geographical location that a user specifies when they wish to obtain information about a particular place.

[0964] "Search results" refer to information extracted from a database of past image data related to the location specified by the user.

[0965] "Filtering" is the process of removing unnecessary data or sensitive information from search results.

[0966] "Indexing" is the process of organizing and indexing data stored in a database so that it can be searched efficiently.

[0967] A "user" is an individual or group that performs operations to obtain information using a system.

[0968] A "server" is a central management system that collects, analyzes, stores, searches, and provides results to users for image data.

[0969] A "terminal" is a device (such as a smartphone, tablet, or PC) that a user uses to access and operate a system.

[0970] The present invention is a system that collects historical image data, organizes it based on metadata, adds location information and stores it in a database, searches based on the user's specified location, and displays the integrated historical image data and current map data. The following describes in detail how the present invention is implemented.

[0971] First, historical image data is collected. The server executes a script that automatically retrieves image data from publicly available databases on the internet. For example, the "Public Image Database API" or the "Image Search API" can be used. The collected image data is temporarily stored in storage (for example, Amazon S3).

[0972] Next, the server analyzes the collected image data using AI image recognition technology (e.g., Google Cloud Vision API). Here, metadata such as the date, location, and event of the image are extracted from the image data. Based on this metadata, the server adds location information (latitude and longitude) to each image. Specifically, location information is obtained using the Google Maps Geocoding API.

[0973] The added location-tagged image data is stored in a database (e.g., PostgreSQL). This database is indexed using Elasticsearch to enable efficient searching later on.

[0974] Users access the system through a dedicated application or website. Users specify a location they want to view on a map and submit a request. This request is received by the server, which searches its database for historical image data corresponding to the specified location. The server then organizes the search results, excluding sensitive information (such as detailed addresses or images of individuals' faces) before sending them to the user's device.

[0975] The device integrates received historical image data and metadata with current map data (e.g., Google Maps API) and displays it on the screen. This allows the user to compare and verify past and present conditions.

[0976] Specific example

[0977] For example, consider a case where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. This information is sent to the server as a result of the user's actions. The server searches its database for entries containing historical image data related to the specified location in Shinjuku Ward and retrieves relevant data such as "a photograph of a residential area in Shinjuku Ward in the 1930s" or "a picture of the area during the disaster recovery period."

[0978] The server filters and organizes this data, then sends the organized data to the user's terminal. The terminal receives the data and displays past images and their descriptions on the screen along with the current map. The user can easily compare the past and the present.

[0979] Examples of prompts to input into a generative AI model

[0980] Example of a prompt:

[0981] Please provide a detailed explanation of how a system works to integrate "photographs of residential areas in Shinjuku Ward from the 1930s" with "a current map of Shinjuku Ward."

[0982] Please describe the specific steps involved in each stage of the process.

[0983] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0984] Step 1:

[0985] The server automatically retrieves image data from publicly available databases on the internet.

[0986] Input: API endpoint for the public image database.

[0987] Data processing / calculation: Send an API request and temporarily store the retrieved image data.

[0988] Output: Temporarily saved image data.

[0989] Specific operation: A Python script is executed to send requests to a public database API and save the retrieved image data to storage such as Amazon S3.

[0990] Step 2:

[0991] The server analyzes the collected image data using AI image recognition technology.

[0992] Input: Temporarily saved image data.

[0993] Data processing / calculation: Send images to an AI image recognition service and extract metadata such as the date and location of the photo, and information about the event.

[0994] Output: Image data with metadata attached.

[0995] Specific operation: Use the Google Cloud Vision API to analyze image data and retrieve metadata including the date and time and location of the image.

[0996] Step 3:

[0997] The server adds latitude and longitude location information to the image based on the metadata.

[0998] Input: Image data with metadata attached.

[0999] Data processing / calculation: Use the Geocoding API to convert text information (address / place name) into latitude and longitude, and add location information to image data.

[1000] Output: Image data with latitude and longitude information added.

[1001] Specific operation: Use the Google Maps Geocoding API to parse the address information in the metadata and obtain the appropriate latitude and longitude location information.

[1002] Step 4:

[1003] The server stores location-tagged image data in a database.

[1004] Input: Image data with latitude and longitude information attached.

[1005] Data processing / calculation: Insert image data into the database and create indexes to enable efficient searching.

[1006] Output: Image data with location information stored in the database.

[1007] Specific operation: Store image data in a PostgreSQL database and perform indexing using Elasticsearch.

[1008] Step 5:

[1009] Users access the system through a dedicated application or website, specify the location they want to view on a map, and submit a request.

[1010] Input: Geographic location information specified by the user.

[1011] Data processing / calculation: Send a request to the server and query information from the specified location.

[1012] Output: Request received by the server.

[1013] Specific operation: The user uses an application or website to click on a location they want to see on a map and sends that information to the server.

[1014] Step 6:

[1015] The server receives a user request and searches the database for historical image data corresponding to the specified location.

[1016] Input: Location request received from the user.

[1017] Data processing / calculation: Search the database for image data corresponding to the queried location and perform filtering.

[1018] Output: Filtered search results.

[1019] Specific operation: Execute Elasticsearch queries to search the database for appropriate image data and exclude sensitive information.

[1020] Step 7:

[1021] The server organizes the search results and sends them to the user's device.

[1022] Input: Filtered search results.

[1023] Data processing / calculation: Organize search results and convert them into a format that can be sent to the user.

[1024] Output: The organized data sent to the user's terminal.

[1025] Specific operation: Convert the search results to JSON format and send them to the user's terminal as an HTTP response.

[1026] Step 8:

[1027] The terminal integrates the received historical image data and metadata with current map data and displays it on the screen.

[1028] Input: Data sent from the server.

[1029] Data Processing / Calculation: Process the received data into a format that can be displayed by integrating it with the map API.

[1030] Output: Integrated data displayed on the user's screen.

[1031] Specific operation: Use the Google Maps API to integrate historical image data with current map data and display it on the user's device.

[1032] (Application Example 1)

[1033] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1034] Conventional technologies had limited means of collecting and effectively providing historical image data to users, making it difficult to compare past and present situations. Furthermore, there was no system that could provide users with relevant contemporary product information based on their interests and enable purchases. Therefore, there is a need to improve the convenience for users when purchasing related products using historical information.

[1035] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1036] In this invention, the server includes means for collecting past image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving a location specification from a user, means for searching the database based on the location specification, means for providing past image data as a search result, means for integrating and displaying the past image data and current map data, means for displaying past and current image data and providing explanatory text, and means for providing related product information based on the user's interests and enabling purchase. This makes it possible for the user to easily compare past and present situations and effectively purchase related products.

[1037] "Past image data" refers to image data taken at a specific time, representing past events or scenes.

[1038] "Metadata" refers to attribute information associated with image data, including information such as the date and time of shooting, location, and events.

[1039] "Location information" refers to geographical information such as latitude and longitude that indicates the location where the image data was taken.

[1040] A "database" is a system that systematically stores and manages digital data, enabling efficient searching.

[1041] "Location specification" is the operation of specifying a particular geographical location that the user is interested in.

[1042] "Searching" is the operation of finding data within a database based on specific criteria.

[1043] "Current map data" refers to digital map data that displays the latest geographical information.

[1044] "Integrated display" means displaying different data together on a single screen or in a single format.

[1045] "Displaying past and present image data" means displaying past image data and current map data on the screen simultaneously.

[1046] "Description" refers to additional information or explanatory text related to image data.

[1047] "Related product information" refers to product data provided based on the user's interests.

[1048] "Means of enabling purchase" refers to a function that allows users to directly purchase products they are interested in.

[1049] This invention is a system that collects historical image data, organizes it based on metadata, adds location information, stores it in a database, searches based on the user's specified location, and displays the integrated historical image data and current map data. Furthermore, it includes a function to provide relevant product information based on the user's interests and enable purchases.

[1050] The server is at the heart of the system and performs the following roles:

[1051] 1. Server

[1052] The server collects historical image data, extracts metadata, adds location information, and stores it in a database. Furthermore, it receives location information from the user, searches the database based on that information, and provides historical image data as search results. The server integrates current map data with historical image data and displays it, providing the user with relevant product information.

[1053] 2. Terminal

[1054] A terminal is a device that users access and operate through a dedicated application. This includes smartphones, tablets, and PCs. The terminal displays historical image data and current map data received from the server, and displays explanatory text and related product information according to user requests.

[1055] 3. User

[1056] Users specify their location through a dedicated interface and view and manipulate data provided by the system. They can compare past and present situations, obtain relevant product information, and purchase items of interest directly.

[1057] Specific example

[1058] The following is a specific example of the process.

[1059] The user launches the "Time Travel Shopping App" on their smartphone and specifies a particular location on the map, for example, "Shinjuku Ward." They also enter a time period for which they want to view past images, such as "1930-1940." This information is sent from the device to the server. The server searches its database based on the specified latitude, longitude, and date range and retrieves the corresponding past image data.

[1060] The acquired image data includes metadata such as the date and location of the photograph, and this information is transmitted to the device along with the image data. The device integrates the past image data with current map data and displays the past and present images simultaneously. A descriptive text related to the image is also displayed.

[1061] Furthermore, the server provides relevant product information based on the user's interests. For example, if an image of a shopping street from the 1930s is displayed, products that were sold at that time and related modern items will be introduced. Users can purchase products they are interested in directly within the app.

[1062] Example of a prompt

[1063] You are an assistant that searches historical image data and integrates it with current map data for display. When given a location name and date range, search for historical image data of that location and display it compared to current map data. Process the following requests:

[1064] Location: Shinjuku Ward

[1065] Start date: 1930-01-01

[1066] End date: 1940-12-31

[1067] This invention allows users to easily access historical information and compare it with the present to gain a deeper historical understanding. Furthermore, by providing related product information, users can proceed directly to the purchase process, significantly improving convenience.

[1068] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1069] Step 1:

[1070] The user specifies a location and time period on a map.

[1071] The user launches the "Time Travel Shopping App" on their smartphone, specifies a location of interest on the map, and enters the period (start and end dates) for which they want to view past images. The entered information is then sent from the device to the server.

[1072] input

[1073] Information about the specified location (e.g., Shinjuku Ward)

[1074] Specified period (start date, end date)

[1075] output

[1076] Request data to the server (location information and time period information)

[1077] Specific actions

[1078] The terminal generates an API request that sends the specified location and time period to the server via the user interface.

[1079] Step 2:

[1080] The server searches for past image data.

[1081] The server searches the database based on the received request data (location information and time period information) to retrieve relevant historical image data. The server uses indexing to perform efficient searches based on location information and time period.

[1082] input

[1083] Request data (location information and time period information)

[1084] output

[1085] Search results (corresponding past image data)

[1086] Specific actions

[1087] The server executes database queries based on the specified latitude, longitude, and date ranges and collects the corresponding image data.

[1088] Step 3:

[1089] The server analyzes the image data and extracts the metadata.

[1090] The server analyzes the acquired image data using AI image recognition technology and extracts metadata for each image, including the date and location of the image, as well as information about related events.

[1091] input

[1092] Search results (past image data)

[1093] output

[1094] Image data with metadata

[1095] Specific actions

[1096] The server uses an AI model (e.g., an image recognition model) to analyze image data and automatically extract metadata such as the date and location of the photograph, and information about the event.

[1097] Step 4:

[1098] The server adds location information and stores it in the database.

[1099] The server adds location information (latitude and longitude) to each image data based on the extracted metadata and stores it in the database as image data with location information.

[1100] input

[1101] Image data with metadata

[1102] output

[1103] Image data with location information

[1104] Specific actions

[1105] The server calculates the latitude and longitude information corresponding to each image based on the metadata, adds location information, and stores it in the database.

[1106] Step 5:

[1107] The server integrates current map data and historical image data and transmits it.

[1108] The server integrates location-based image data with current map data, converts it into a data format that allows the user to compare past images with the current map, and sends it to the terminal.

[1109] input

[1110] Image data with location information

[1111] Current map data

[1112] output

[1113] Integrated data (data from past images and current maps)

[1114] Specific actions

[1115] The server overlays historical images onto map data based on location information and converts them into a format that allows for comparison and display of both.

[1116] Step 6:

[1117] The device displays integrated data and provides explanatory text.

[1118] The device displays the received integrated data and compares past and present images. It also displays explanatory text related to the images.

[1119] input

[1120] Integrated data (data from past images and current maps)

[1121] output

[1122] Images and descriptions displayed on the user interface

[1123] Specific actions

[1124] The device displays integrated data in a user-friendly format and provides a convenient interface for comparing past and present situations.

[1125] Step 7:

[1126] The server provides relevant product information based on the user's interests.

[1127] The server generates relevant product information based on past image data, descriptions, and user operation data, and transmits this information to the terminal.

[1128] input

[1129] User operation data

[1130] Past image data

[1131] Description

[1132] output

[1133] Related product information

[1134] Specific actions

[1135] The server uses a generative AI model to analyze user interests and generate data to select and provide corresponding product information.

[1136] Step 8:

[1137] The device displays related product information and enables the purchase process.

[1138] The device displays the received related product information, allowing the user to purchase items they are interested in directly.

[1139] input

[1140] Related product information

[1141] output

[1142] Product information and purchase options displayed on the user interface

[1143] Specific actions

[1144] The terminal displays related product information and provides an interface that allows users to proceed directly to the purchase process by clicking on a product.

[1145] The above outlines the specific processing steps. This processing flow allows users to easily compare past and current situations and gain an effective experience in purchasing related products.

[1146] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1147] This invention combines a system that collects historical image data, organizes it based on metadata, stores it in a database with location information, searches it based on the user's specified location, and displays the integrated historical image data and current map data with an emotion engine that recognizes the user's emotions. This enables the provision of optimal information according to the user's emotional state.

[1148] System Configuration

[1149] 1. Server: This server is central to the present invention, performing data processing and storage, and responding to user requests. It also incorporates an emotion engine to analyze user emotions.

[1150] 2. Device: A device (such as a smartphone, tablet, or PC) that a user uses to access and operate the system through an application or website.

[1151] 3. User: A person who specifies a location, views the results, and performs operations through a dedicated interface.

[1152] System operation

[1153] First, historical image data is collected. The server executes a script that automatically retrieves image data from publicly available databases on the internet. The collected image data is temporarily stored in storage.

[1154] Next, the server analyzes the collected image data using AI image recognition technology. The AI ​​extracts metadata from the images, including the date, location, and events that occurred. Based on this metadata, the server adds location information (latitude and longitude) to each image.

[1155] The image data with added location information is stored in a specialized database. This database is indexed later to enable efficient searching.

[1156] Users access the system through a dedicated application or website. Users specify a location they want to view on a map and submit a request. The server receives this request and searches its database for historical image data corresponding to the specified location.

[1157] The server organizes the search results and prepares image datasets related to the specified locations. It filters out any sensitive information to ensure that the images are suitable for publication.

[1158] This is where the emotion engine comes in. The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions in real time. Emotions are categorized into positive, negative, and neutral.

[1159] The server optimizes the information it provides based on the user's emotional state. For example, if a user is showing negative emotions, it prioritizes providing image data with generally positive content. It also performs filtering corresponding to specific emotional states to prevent the display of information that may be offensive to the user.

[1160] Finally, the server sends the filtered data to the user's terminal. The terminal analyzes the received data and displays the historical image data and current map data integrated on the interface. The user can simultaneously view historical and current information for a specified location.

[1161] Specific example

[1162] For example, consider a scenario where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. Meanwhile, the emotion engine recognizes "positive emotions" from the user's facial expressions and voice. This information is sent to the server.

[1163] The server searches the database for entries containing historical image data related to a specified location in Shinjuku Ward, retrieving entries such as "a photograph of a residential area in Shinjuku Ward in the 1930s" or "positive anecdotes from the reconstruction period after the Great Kanto Earthquake." Simultaneously, the sentiment engine optimizes the data to provide an overall positive content, taking into account the user's positive emotions.

[1164] The server filters this data, excluding sensitive information, and then sends the organized data to the user's device. The device receives the data and displays past images and their descriptions on the screen along with a current map. Users can easily compare the past and present while prioritizing the viewing of positive information.

[1165] Thus, the present invention is a system that efficiently links past image data and current map data, and further utilizes an emotion engine to provide optimal information tailored to the user's emotional state.

[1166] The following describes the processing flow.

[1167] Step 1:

[1168] Server: Executes scripts to access publicly available databases and libraries on the internet and collect historical image data. The collected image data is stored in temporary storage.

[1169] Step 2:

[1170] Server: The server initiates analysis of the collected image data using AI image recognition technology. The AI ​​model extracts metadata from the images, including the date, location, and events that occurred.

[1171] Step 3:

[1172] Server: Based on the extracted metadata, it adds location information such as latitude and longitude to each image. This information is used to set location attributes for the image data.

[1173] Step 4:

[1174] Server: Stores location-tagged image data in a database. An index is created during storage to enable efficient searching later.

[1175] Step 5:

[1176] User: Access the dedicated application or website and log in. The user uses the map interface to specify the location for which they want to view past information by clicking or tapping.

[1177] Step 6:

[1178] Terminal: Sends the user's location information as a request to the server. The request includes the latitude and longitude information of the specified location.

[1179] Step 7:

[1180] Server: Based on the received location request, it searches the database. It retrieves historical image data and its metadata associated with the specified latitude and longitude.

[1181] Step 8:

[1182] Server: Organizes search results and prepares image datasets related to the specified locations. Filters out sensitive information to ensure it is appropriate for publication.

[1183] Step 9:

[1184] Server: Sends filtered data to the user's terminal. The data is sent in a structured format such as JSON.

[1185] Step 10:

[1186] Terminal: Analyzes received data and integrates historical image data and current map data on the interface for display. Users can simultaneously view historical and current information for a specified location.

[1187] Step 11:

[1188] User: Uses the device's camera and microphone to express emotions through facial expressions and voice. The emotion engine acquires this data and analyzes emotions in real time.

[1189] Step 12:

[1190] Server: The emotion engine classifies the user's emotions as positive, negative, or neutral, and provides optimized information based on that emotional state.

[1191] Step 13:

[1192] Server: Filters relevant metadata based on emotional state. For example, if a user is expressing negative emotions, it prioritizes providing image data with positive content.

[1193] Step 14:

[1194] Server: Retransmits the information set optimized by the emotion engine to the user's terminal.

[1195] Step 15:

[1196] Device: Redisplays past images and their descriptions that match the user's emotional state, along with the current map. Users can enjoy viewing information in an emotionally sensitive way.

[1197] This series of steps allows users to view historical image data and current map data in an integrated manner, and then receive optimized information from the emotion engine.

[1198] (Example 2)

[1199] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1200] Conventional systems that collect historical image data, add location information, store it in a database, and search based on the user's specified location do not consider the user's emotional state, and the information provided may not always be optimal. Furthermore, if sensitive information is not filtered, inappropriate information may be provided. To solve this problem, a system is needed that provides optimal information tailored to the user's emotional state.

[1201] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting past image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the image data with location information in a database, means for receiving a location specification from the user, means for searching the database based on the location specification, means for providing past image data as a search result, means for integrating and displaying past image data and current map data, and an emotion engine that recognizes the user's emotions, analyzes the user's emotions using the emotion engine, and optimizes the information to be provided based on the analysis results. This makes it possible to provide optimal information according to the user's emotional state.

[1202] "Past image data" refers to photographs and image files taken in the past, including data such as the date, time, and location of the shooting.

[1203] "Metadata" refers to supplementary information related to image data, including data such as the date and time of shooting, location, and details of the event.

[1204] "Location information" refers to longitude and latitude data used to indicate a specific location.

[1205] A "database" refers to a specialized data storage system for efficiently storing, searching, and managing image data, its metadata, and location information.

[1206] A "user" refers to a person who specifies a location or performs other operations through the system.

[1207] "Location specification" refers to the user selecting a location they want to see on a map or specifying it in text.

[1208] An "emotion engine" refers to a system equipped with technology that analyzes a user's emotions in real time from their facial expressions and voice, and classifies that emotional state as positive, negative, neutral, etc.

[1209] "Integration" refers to combining historical image data and current map data and displaying them on a single interface.

[1210] "Optimization" refers to adjusting the content and order of information provided based on the user's emotional state, as analyzed by the emotion engine, to make the user experience as effective as possible.

[1211] "Sensitive information" refers to information that should be restricted from public disclosure based on specific criteria, such as personal information or data deemed socially inappropriate.

[1212] Modes for carrying out the invention

[1213] This invention combines a system that collects historical image data, organizes it based on metadata, stores it in a database with location information, searches it based on the user's specified location, and displays the integrated historical image data and current map data with an emotion engine that recognizes the user's emotions. This enables the provision of optimal information according to the user's emotional state.

[1214] Hardware and software configuration

[1215] 1. Server:

[1216] The server performs the central data processing and storage of this invention, as well as responding to user requests. It also incorporates an emotion engine to analyze user emotions. This requires a high-performance server, utilizing software such as Python, MySQL, and OpenCV.

[1217] 2. Terminal:

[1218] A terminal is a device that a user uses to access and operate a system through an application or website. Typically, smartphones, tablets, and PCs are used.

[1219] 3. User:

[1220] A user is a person who specifies a location, views the results, and performs actions through a dedicated interface.

[1221] Program Processing Description

[1222] First, historical image data is collected. The server executes a script to automatically retrieve image data from publicly available databases on the internet. The collected image data is temporarily stored in the server's storage. Next, the server analyzes the collected image data using AI image recognition technology (e.g., OpenCV models). The AI ​​extracts metadata from the images, such as the date and location of the image, and events, and adds location information (latitude and longitude) based on this metadata. The image data with location information is stored in a specialized database, such as a MySQL database, and indexed to enable efficient searching.

[1223] Users access the system through a dedicated application or website. Users specify a location they wish to view on a map and submit a request. The server receives this request and searches its database for historical image data corresponding to the specified location. The server organizes the search results and prepares an image dataset related to the specified location. During this process, it filters out sensitive information and verifies whether the data is appropriate for publication.

[1224] The addition of an emotion engine allows the system to analyze the user's facial expressions and voice to recognize their emotions in real time. Emotions are categorized as positive, negative, and neutral, and the server optimizes the information it provides based on the user's emotional state. For example, if a user is showing negative emotions, the system prioritizes providing image data with generally positive content. It also filters content based on specific emotional states to prevent the display of information that may be unpleasant to the user.

[1225] Finally, the server sends the filtered data to the user's terminal. The terminal analyzes the received data and can integrate and display historical image data and current map data on its interface. The user can simultaneously view historical and current information for a specified location.

[1226] Specific example

[1227] For example, consider a scenario where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. Meanwhile, the emotion engine recognizes "positive emotions" from the user's facial expressions and voice. This information is sent to the server. The server searches its database for entries containing historical image data related to the specified location in Shinjuku Ward, retrieving, for example, "a photograph of a residential area in Shinjuku Ward in the 1930s" or "positive episodes from the reconstruction period after the Great Kanto Earthquake." At the same time, the emotion engine optimizes the data to provide overall positive content, taking into account the user's positive emotions. The server filters this data, excluding sensitive information, and then sends the organized data to the user's device. The device receives the data and displays the historical images and their descriptions on the screen along with the current map. The user can easily compare the past and present while prioritizing the viewing of positive information.

[1228] Thus, the present invention is a system that efficiently links past image data and current map data, and further utilizes an emotion engine to provide optimal information tailored to the user's emotional state.

[1229] Example of a prompt

[1230] The following are examples of prompt statements to input into the generating AI model.

[1231] Example 1:

[1232] Search for image data of "Shinjuku Ward, 1930s residential area" and provide information that includes positive episodes. The user's emotional state is positive.

[1233] Example 2:

[1234] Display image data related to the "episode from the reconstruction period after the Great Kanto Earthquake" specified by the user. The emotion engine will analyze the user's emotional state, and if it indicates positive emotions, prioritize providing data with positive content.

[1235] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1236] Step 1:

[1237] Image data collection:

[1238] The server executes a script to automatically retrieve image data from a publicly available database on the internet. Specifically, it uses the Python requests library to send an HTTP request from a specific URL and download the image data. The input for this step is the URL of the public database, and the output is the downloaded image file.

[1239] Specific examples of operation:

[1240] The server accesses https: / / example.com / api / images and downloads the image data.

[1241] Step 2:

[1242] Image data analysis:

[1243] The server analyzes the collected image data using AI image recognition technology (e.g., OpenCV models). The input is the downloaded image data, and the output is metadata about the date, location, and event of the photograph. The AI ​​extracts this metadata from text information and digital prints within the image.

[1244] Specific examples of operation:

[1245] The server uses OpenCV and pytesseract to analyze image data and extract text information.

[1246] Step 3:

[1247] Adding location information:

[1248] The server uses the extracted metadata to add location information (latitude and longitude) to the image data. The input is metadata about the shooting location, and the output is location-attached image data containing latitude and longitude information. In this step, the metadata is sent to a geocoding API to obtain the latitude and longitude.

[1249] Specific examples of operation:

[1250] The server sends location information for the filming location in Shinjuku Ward to a geocoding API to obtain the latitude and longitude.

[1251] Step 4:

[1252] Saving to the database:

[1253] The server stores location-tagged image data in a specialized database. The input is location-tagged image data, and the output is the data entries stored in the database. For storage efficiency, the server stores the image data itself, its links, metadata, location information, etc.

[1254] Specific examples of operation:

[1255] The server connects to a MySQL database and stores image data, including location information and metadata.

[1256] Step 5:

[1257] Receiving user requests:

[1258] Users operating the terminal access the system through a dedicated application or website, specify the location they want to view on a map, and send a request. The input is the location specified by the user, and the output is a search request related to that location.

[1259] Specific examples of operation:

[1260] The user launches the application, clicks on a location in Shinjuku Ward on the map, and sends a request.

[1261] Step 6:

[1262] Searching and filtering data:

[1263] The server searches its database for historical image data corresponding to a specified location based on the received user request. The input is the user's request, and the output is the associated image dataset. Simultaneously, sensitive information is filtered.

[1264] Specific examples of operation:

[1265] The server searches the database for entries related to Shinjuku Ward, filters out sensitive information, and retrieves the results.

[1266] Step 7:

[1267] Emotion recognition by an emotion engine:

[1268] The user's facial expressions and voice are collected through the device's camera and microphone, and the server's emotion engine analyzes them in real time. The input is the user's facial expressions and voice data, and the output is categorized as positive, negative, or neutral emotions.

[1269] Specific examples of operation:

[1270] The server uses EmotionEngine to analyze the user's real-time data and recognizes their emotional state as positive.

[1271] Step 8:

[1272] Optimizing the information provided:

[1273] The server optimizes the information it provides based on the user's emotional state. The input consists of emotion recognition results and historical image data, while the output is an optimized image dataset. For example, if a negative emotion is indicated, positive image data is prioritized.

[1274] Specific examples of operation:

[1275] The server considers the emotional state to be positive and filters out data that includes positive episodes.

[1276] Step 9:

[1277] Sending and displaying data:

[1278] The server sends optimized data to the user's terminal. The input is optimized image data, and the output is data transmission to the terminal. The terminal analyzes the received data and displays it on the interface, integrating historical image data and current map data. This allows the user to view historical and current information for a specified location simultaneously.

[1279] Specific examples of operation:

[1280] The server sends optimized image data to the user's terminal, which then displays it on the screen along with the current map data.

[1281] (Application Example 2)

[1282] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[1283] This invention relates to a system that collects and organizes past image data and searches and displays it based on a location specified by the user. However, conventional systems provide uniform information without considering the user's emotional state, resulting in the inability to provide information tailored to the individual user's emotions and needs. As a result, the user experience may be compromised. This invention aims to solve this problem and realize the provision of optimal information according to the user's emotional state.

[1284] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1285] In this invention, the server includes means for collecting past image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving a location specification from a user, means for searching the database based on the location specification, means for providing past image data as a search result, means for integrating and displaying the past image data and current map data, and means for analyzing the user's emotions and optimizing the image data provided based on those emotions. This makes it possible to provide information that takes into account the user's emotional state.

[1286] "Past image data" refers to digital data of photographs and videos taken in the past.

[1287] "Metadata" refers to information associated with image data, including details about the date and location of the photograph, and events that occurred.

[1288] "Location information" refers to data that indicates a geographical location, and is usually represented by latitude and longitude.

[1289] A "database" is a system designed to efficiently store and retrieve large amounts of data.

[1290] "Location specification" refers to the operation in which a user selects or specifies a particular geographical location.

[1291] "Map data" refers to digital data containing geographical information, showing current topography and the locations of buildings.

[1292] "Emotion analysis" is a technology that identifies emotions from a user's facial expressions and voice, and analyzes their emotional state.

[1293] "Searching" is the process of exploring a database based on specified conditions and extracting relevant data.

[1294] "Integrated display" is the operation of displaying multiple data points together on a single screen.

[1295] "Optimization" means adjusting or improving something to best suit a specific purpose.

[1296] This invention combines a system that collects historical image data, organizes it based on metadata, stores it in a database with location information, searches it based on the user's specified location, and displays the integrated historical image data and current map data with an emotion engine that recognizes the user's emotions. This enables the provision of optimal information according to the user's emotional state.

[1297] The system's core is the server. The server handles data processing, storage, and responding to user requests. It also incorporates an emotion recognition engine to analyze user emotions. Specifically, the server includes the following:

[1298] 1. Collection of image data

[1299] The server executes a script to automatically retrieve image data from publicly available databases on the internet. This temporarily stores past image data in storage.

[1300] 2. Extraction of metadata

[1301] The server analyzes the collected image data using AI image recognition technology (specifically, TensorFlow and Keras can be used). The AI ​​extracts metadata from the images, including the date, location, and events that occurred. Based on this metadata, the server adds location information (latitude and longitude) to each image.

[1302] 3. Storing and retrieving data in the database

[1303] The location-tagged image data is stored in a specialized database (MongoDB or PostgreSQL are examples). This database is indexed to enable efficient searching later. Based on the location specified by the user, the server searches the database and retrieves historical image data corresponding to the specified location.

[1304] 4. Emotion Recognition and Optimization

[1305] An emotion recognition engine (specifically, Microsoft Azure's Emotion API or Google Cloud AI's Vision AI can be used) analyzes the user's facial expressions and voice to recognize their emotions in real time. Emotions are categorized as positive, negative, and neutral. The server optimizes the information it provides based on the user's emotional state. For example, it prioritizes providing positive image data to a user exhibiting negative emotions.

[1306] 5. Integrated display

[1307] The server sends filtered data to the user's device (smart glasses, smartphone, etc.). The device analyzes the received data and displays it by integrating historical image data and current map data on its interface. For example, map data can be displayed using the Google Maps API or Apple Maps.

[1308] Specific example

[1309] For example, consider a scenario where a user specifies a tourist spot in Shinjuku Ward. The user launches a dedicated application and selects a specific location in Shinjuku Ward on a map. Simultaneously, the user's facial expressions and voice are analyzed via camera and microphone, and an emotion engine recognizes "positive emotions." This information is then sent to the server.

[1310] The server searches its database for historical image data related to a specified location in Shinjuku Ward (e.g., photos of Shinjuku Ward from the 1930s or positive anecdotes) and provides it. The sentiment engine prioritizes selecting data with an overall positive content, taking into account that the user has expressed positive emotions.

[1311] The server filters this data, excluding sensitive information, and then sends the organized data to the user's device. The user's device receives the data and displays past images and their descriptions on the screen along with the current map. In this way, the user can compare the past and present while prioritizing the viewing of positive information.

[1312] Example of a prompt

[1313] "Integrate historical image data with current map data and provide information that matches the user's positive emotions. For example, if a user is moved, display historically positive photos or stories about that place."

[1314] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1315] Step 1:

[1316] Users access the system through smart glasses or a smartphone app and specify a particular geographical location. This input includes information about the geographical location specified by the user (latitude and longitude).

[1317] Specific operation: The user launches a map application and taps or clicks on a specific location in Shinjuku Ward on the map to specify it.

[1318] Step 2:

[1319] The server receives location information sent by the user and searches the database for historical image data associated with that location. The input here is the geographical location information specified by the user, and the output is the corresponding historical image data.

[1320] Specific operation: The server generates a database query to search for image data corresponding to the specified latitude and longitude. MongoDB or PostgreSQL can be used as the database.

[1321] Step 3:

[1322] The server analyzes the collected image data using AI image recognition technology and extracts metadata (date of capture, location, and event). The input here is historical image data, and the output is the associated metadata.

[1323] Specific operation: As AI image recognition technology, TensorFlow and Keras are used to extract necessary information from image data.

[1324] Step 4:

[1325] The server analyzes the user's emotions using data acquired from the user's smart glasses or smartphone camera / microphone. The input is the user's facial expression images and audio data, and the output is identified emotion information.

[1326] Specific operation: An emotion recognition engine (e.g., Microsoft Azure's Emotion API) is used to analyze the user's facial expressions and voice to determine the category of emotion (positive, negative, neutral).

[1327] Step 5:

[1328] The server selects the most suitable image data based on the analyzed sentiment information. The input here is image data with sentiment information and metadata, and the output is historical image data optimized for the user's sentiment.

[1329] Specific operation: The server refers to emotional information and, for example, prioritizes selecting bright and hopeful images for users who exhibit positive emotions.

[1330] Step 6:

[1331] The server sends image data optimized according to the user's emotions to the user's device. Here, the input is the optimized image data, and the output is the content displayed on the user's device.

[1332] Specific operation: The server organizes the selected image data, bundles it into a format such as JSON, and sends it to the user's smart glasses or smartphone application.

[1333] Step 7:

[1334] The terminal integrates and displays historical image data and current map data based on the received data. The input consists of image data and map data sent from the server, and the output is an integrated display screen that the user can view.

[1335] Specific operation: The device will use the Google Maps API or Apple Maps to build an interface that overlays current map data with historical image data.

[1336] Through the above processing steps, users can simultaneously compare the past and present of their location while receiving information optimized for their emotions at that time.

[1337] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1338] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1339] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[1340] [Fourth Embodiment]

[1341] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[1342] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1343] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1344] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[1345] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[1346] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[1347] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[1348] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[1349] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[1350] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1351] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1352] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[1353] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1354] The present invention is a system that collects historical image data, organizes it based on metadata, adds location information and stores it in a database, searches based on the user's specified location, and displays the integrated historical image data and current map data. The following describes in detail how the present invention is implemented.

[1355] System Configuration

[1356] 1. Server: This server is central to the present invention, performing data processing and storage, and responding to user requests.

[1357] 2. Device: A device (such as a smartphone, tablet, or PC) that a user uses to access and operate the system through an application or website.

[1358] 3. User: A person who specifies a location, views the results, and performs operations through a dedicated interface.

[1359] System operation

[1360] First, historical image data is collected. The server executes a script that automatically retrieves image data from publicly available databases on the internet. The collected image data is temporarily stored in storage.

[1361] Next, the server analyzes the collected image data using AI image recognition technology. The AI ​​extracts metadata from the images, including the date, location, and events that occurred. Based on this metadata, the server adds location information (latitude and longitude) to each image.

[1362] The image data with added location information is stored in a specialized database. This database is indexed later to enable efficient searching.

[1363] Users access the system through a dedicated application or website. Users specify a location they want to view on a map and submit a request. The server receives this request and searches its database for historical image data corresponding to the specified location.

[1364] The server organizes the search results and sends them to the user's terminal. The terminal integrates the received historical image data and metadata with current map data and displays it on the screen. This allows the user to compare and verify past and present conditions.

[1365] Specific example

[1366] For example, consider a case where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. This information is sent to the server by the user's action. The server searches its database for entries containing historical image data related to the specified location in Shinjuku Ward, and retrieves relevant data such as "a photograph of a residential area in Shinjuku Ward in the 1930s" or "a depiction of the area during the reconstruction period after the Great Kanto Earthquake."

[1367] The server filters this data, removing sensitive information (such as detailed addresses or images of individuals' faces), and then sends the organized data to the user's device. The device receives the data and displays past images and their descriptions on the screen along with a current map. The user can easily compare the past and present.

[1368] Thus, the present invention is a system that efficiently links historical image data and current map data to provide useful information to users.

[1369] The following describes the processing flow.

[1370] Step 1:

[1371] Server: Executes scripts to access publicly available databases and libraries on the internet and collect historical image data. The collected image data is stored in temporary storage.

[1372] Step 2:

[1373] Server: The server initiates analysis of the collected image data using AI image recognition technology. The AI ​​model extracts metadata from the images, including the date, location, and events that occurred.

[1374] Step 3:

[1375] Server: Based on the extracted metadata, it adds location information such as latitude and longitude to each image. This information is used to set location attributes for the image data.

[1376] Step 4:

[1377] Server: Stores location-tagged image data in a database. An index is created during storage to enable efficient searching later.

[1378] Step 5:

[1379] User: Access the dedicated application or website and log in. The user uses the map interface to specify the location for which they want to view past information by clicking or tapping.

[1380] Step 6:

[1381] Terminal: Sends the user's location information as a request to the server. The request includes the latitude and longitude information of the specified location.

[1382] Step 7:

[1383] Server: Based on the received location request, it searches the database. It retrieves historical image data and its metadata associated with the specified latitude and longitude.

[1384] Step 8:

[1385] Server: Organizes search results and prepares image datasets related to the specified locations. Filters out sensitive information to ensure it is appropriate for publication.

[1386] Step 9:

[1387] Server: Sends filtered data to the user's terminal. The data is sent in a structured format such as JSON.

[1388] Step 10:

[1389] Terminal: Analyzes received data and integrates historical image data and current map data on the interface for display. Users can simultaneously view historical and current information for a specified location.

[1390] Step 11:

[1391] User: Based on the displayed information, users can view more detailed information as needed or based on their interests. Users can also specify other locations and check past information using a similar procedure.

[1392] This series of processes allows users to view historical image data and current map data in an integrated manner, making it easy to understand the historical background and past conditions of a location.

[1393] (Example 1)

[1394] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1395] Conventional technologies had problems with efficiency and accuracy in systems that properly collected, analyzed, and stored historical image data and linked it with location information for searching. Furthermore, the process of integrating historical image data corresponding to a user-specified region with current map data presented challenges in filtering search results and efficiently managing the data.

[1396] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[1397] In this invention, the server includes means for collecting historical image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving a location specification from a user, means for searching the database based on the location specification, means for providing historical image data as search results, means for integrating and displaying the historical image data and current map data, means for filtering the search results, and means for performing indexing. This makes it possible to efficiently and accurately collect and store historical image data and provide appropriate search results in response to user requests.

[1398] "Image data" refers to digital image information that records past events and places.

[1399] "Metadata" refers to information associated with the image data itself, including data such as the date and time of shooting, the location of shooting, and details of the event.

[1400] "Location information" refers to digital data of latitude and longitude used to indicate a specific location.

[1401] A "database" is an information management system for efficiently storing, managing, and retrieving collected image data and metadata.

[1402] "Location specification" refers to the geographical location that a user specifies when they wish to obtain information about a particular place.

[1403] "Search results" refer to information extracted from a database of past image data related to the location specified by the user.

[1404] "Filtering" is the process of removing unnecessary data or sensitive information from search results.

[1405] "Indexing" is the process of organizing and indexing data stored in a database so that it can be searched efficiently.

[1406] A "user" is an individual or group that performs operations to obtain information using a system.

[1407] A "server" is a central management system that collects, analyzes, stores, searches, and provides results to users for image data.

[1408] A "terminal" is a device (such as a smartphone, tablet, or PC) that a user uses to access and operate a system.

[1409] The present invention is a system that collects historical image data, organizes it based on metadata, adds location information and stores it in a database, searches based on the user's specified location, and displays the integrated historical image data and current map data. The following describes in detail how the present invention is implemented.

[1410] First, historical image data is collected. The server executes a script that automatically retrieves image data from publicly available databases on the internet. For example, the "Public Image Database API" or the "Image Search API" can be used. The collected image data is temporarily stored in storage (for example, Amazon S3).

[1411] Next, the server analyzes the collected image data using AI image recognition technology (e.g., Google Cloud Vision API). Here, metadata such as the date, location, and event of the image are extracted from the image data. Based on this metadata, the server adds location information (latitude and longitude) to each image. Specifically, location information is obtained using the Google Maps Geocoding API.

[1412] The added location-tagged image data is stored in a database (e.g., PostgreSQL). This database is indexed using Elasticsearch to enable efficient searching later on.

[1413] Users access the system through a dedicated application or website. Users specify a location they want to view on a map and submit a request. This request is received by the server, which searches its database for historical image data corresponding to the specified location. The server then organizes the search results, excluding sensitive information (such as detailed addresses or images of individuals' faces) before sending them to the user's device.

[1414] The device integrates received historical image data and metadata with current map data (e.g., Google Maps API) and displays it on the screen. This allows the user to compare and verify past and present conditions.

[1415] Specific example

[1416] For example, consider a case where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. This information is sent to the server as a result of the user's actions. The server searches its database for entries containing historical image data related to the specified location in Shinjuku Ward and retrieves relevant data such as "a photograph of a residential area in Shinjuku Ward in the 1930s" or "a picture of the area during the disaster recovery period."

[1417] The server filters and organizes this data, then sends the organized data to the user's terminal. The terminal receives the data and displays past images and their descriptions on the screen along with the current map. The user can easily compare the past and the present.

[1418] Examples of prompts to input into a generative AI model

[1419] Example of a prompt:

[1420] Please provide a detailed explanation of how a system works to integrate "photographs of residential areas in Shinjuku Ward from the 1930s" with "a current map of Shinjuku Ward."

[1421] Please describe the specific steps involved in each stage of the process.

[1422] The flow of the specific processing in Example 1 will be explained using Figure 11.

[1423] Step 1:

[1424] The server automatically retrieves image data from publicly available databases on the internet.

[1425] Input: API endpoint for the public image database.

[1426] Data processing / calculation: Send an API request and temporarily store the retrieved image data.

[1427] Output: Temporarily saved image data.

[1428] Specific operation: A Python script is executed to send requests to a public database API and save the retrieved image data to storage such as Amazon S3.

[1429] Step 2:

[1430] The server analyzes the collected image data using AI image recognition technology.

[1431] Input: Temporarily saved image data.

[1432] Data processing / calculation: Send images to an AI image recognition service and extract metadata such as the date and location of the photo, and information about the event.

[1433] Output: Image data with metadata attached.

[1434] Specific operation: Use the Google Cloud Vision API to analyze image data and retrieve metadata including the date and time and location of the image.

[1435] Step 3:

[1436] The server adds latitude and longitude location information to the image based on the metadata.

[1437] Input: Image data with metadata attached.

[1438] Data processing / calculation: Use the Geocoding API to convert text information (address / place name) into latitude and longitude, and add location information to image data.

[1439] Output: Image data with latitude and longitude information added.

[1440] Specific operation: Use the Google Maps Geocoding API to parse the address information in the metadata and obtain the appropriate latitude and longitude location information.

[1441] Step 4:

[1442] The server stores location-tagged image data in a database.

[1443] Input: Image data with latitude and longitude information attached.

[1444] Data processing / calculation: Insert image data into the database and create indexes to enable efficient searching.

[1445] Output: Image data with location information stored in the database.

[1446] Specific operation: Store image data in a PostgreSQL database and perform indexing using Elasticsearch.

[1447] Step 5:

[1448] Users access the system through a dedicated application or website, specify the location they want to view on a map, and submit a request.

[1449] Input: Geographic location information specified by the user.

[1450] Data processing / calculation: Send a request to the server and query information from the specified location.

[1451] Output: Request received by the server.

[1452] Specific operation: The user uses an application or website to click on a location they want to see on a map and sends that information to the server.

[1453] Step 6:

[1454] The server receives a user request and searches the database for historical image data corresponding to the specified location.

[1455] Input: Location request received from the user.

[1456] Data processing / calculation: Search the database for image data corresponding to the queried location and perform filtering.

[1457] Output: Filtered search results.

[1458] Specific operation: Execute Elasticsearch queries to search the database for appropriate image data and exclude sensitive information.

[1459] Step 7:

[1460] The server organizes the search results and sends them to the user's device.

[1461] Input: Filtered search results.

[1462] Data processing / calculation: Organize search results and convert them into a format that can be sent to the user.

[1463] Output: The organized data sent to the user's terminal.

[1464] Specific operation: Convert the search results to JSON format and send them to the user's terminal as an HTTP response.

[1465] Step 8:

[1466] The terminal integrates the received historical image data and metadata with current map data and displays it on the screen.

[1467] Input: Data sent from the server.

[1468] Data Processing / Calculation: Process the received data into a format that can be displayed by integrating it with the map API.

[1469] Output: Integrated data displayed on the user's screen.

[1470] Specific operation: Use the Google Maps API to integrate historical image data with current map data and display it on the user's device.

[1471] (Application Example 1)

[1472] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1473] Conventional technologies had limited means of collecting and effectively providing historical image data to users, making it difficult to compare past and present situations. Furthermore, there was no system that could provide users with relevant contemporary product information based on their interests and enable purchases. Therefore, there is a need to improve the convenience for users when purchasing related products using historical information.

[1474] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[1475] In this invention, the server includes means for collecting past image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving a location specification from a user, means for searching the database based on the location specification, means for providing past image data as a search result, means for integrating and displaying the past image data and current map data, means for displaying past and current image data and providing explanatory text, and means for providing related product information based on the user's interests and enabling purchase. This makes it possible for the user to easily compare past and present situations and effectively purchase related products.

[1476] "Past image data" refers to image data taken at a specific time, representing past events or scenes.

[1477] "Metadata" refers to attribute information associated with image data, including information such as the date and time of shooting, location, and events.

[1478] "Location information" refers to geographical information such as latitude and longitude that indicates the location where the image data was taken.

[1479] A "database" is a system that systematically stores and manages digital data, enabling efficient searching.

[1480] "Location specification" is the operation of specifying a particular geographical location that the user is interested in.

[1481] "Searching" is the operation of finding data within a database based on specific criteria.

[1482] "Current map data" refers to digital map data that displays the latest geographical information.

[1483] "Integrated display" means displaying different data together on a single screen or in a single format.

[1484] "Displaying past and present image data" means displaying past image data and current map data on the screen simultaneously.

[1485] "Description" refers to additional information or explanatory text related to image data.

[1486] "Related product information" refers to product data provided based on the user's interests.

[1487] "Means of enabling purchase" refers to a function that allows users to directly purchase products they are interested in.

[1488] This invention is a system that collects historical image data, organizes it based on metadata, adds location information, stores it in a database, searches based on the user's specified location, and displays the integrated historical image data and current map data. Furthermore, it includes a function to provide relevant product information based on the user's interests and enable purchases.

[1489] The server is at the heart of the system and performs the following roles:

[1490] 1. Server

[1491] The server collects historical image data, extracts metadata, adds location information, and stores it in a database. Furthermore, it receives location information from the user, searches the database based on that information, and provides historical image data as search results. The server integrates current map data with historical image data and displays it, providing the user with relevant product information.

[1492] 2. Terminal

[1493] A terminal is a device that users access and operate through a dedicated application. This includes smartphones, tablets, and PCs. The terminal displays historical image data and current map data received from the server, and displays explanatory text and related product information according to user requests.

[1494] 3. User

[1495] Users specify their location through a dedicated interface and view and manipulate data provided by the system. They can compare past and present situations, obtain relevant product information, and purchase items of interest directly.

[1496] Specific example

[1497] The following is a specific example of the process.

[1498] The user launches the "Time Travel Shopping App" on their smartphone and specifies a particular location on the map, for example, "Shinjuku Ward." They also enter a time period for which they want to view past images, such as "1930-1940." This information is sent from the device to the server. The server searches its database based on the specified latitude, longitude, and date range and retrieves the corresponding past image data.

[1499] The acquired image data includes metadata such as the date and location of the photograph, and this information is transmitted to the device along with the image data. The device integrates the past image data with current map data and displays the past and present images simultaneously. A descriptive text related to the image is also displayed.

[1500] Furthermore, the server provides relevant product information based on the user's interests. For example, if an image of a shopping street from the 1930s is displayed, products that were sold at that time and related modern items will be introduced. Users can purchase products they are interested in directly within the app.

[1501] Example of a prompt

[1502] You are an assistant that searches historical image data and integrates it with current map data for display. When given a location name and date range, search for historical image data of that location and display it compared to current map data. Process the following requests:

[1503] Location: Shinjuku Ward

[1504] Start date: 1930-01-01

[1505] End date: 1940-12-31

[1506] This invention allows users to easily access historical information and compare it with the present to gain a deeper historical understanding. Furthermore, by providing related product information, users can proceed directly to the purchase process, significantly improving convenience.

[1507] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[1508] Step 1:

[1509] The user specifies a location and time period on a map.

[1510] The user launches the "Time Travel Shopping App" on their smartphone, specifies a location of interest on the map, and enters the period (start and end dates) for which they want to view past images. The entered information is then sent from the device to the server.

[1511] input

[1512] Information about the specified location (e.g., Shinjuku Ward)

[1513] Specified period (start date, end date)

[1514] output

[1515] Request data to the server (location information and time period information)

[1516] Specific actions

[1517] The terminal generates an API request that sends the specified location and time period to the server via the user interface.

[1518] Step 2:

[1519] The server searches for past image data.

[1520] The server searches the database based on the received request data (location information and time period information) to retrieve relevant historical image data. The server uses indexing to perform efficient searches based on location information and time period.

[1521] input

[1522] Request data (location information and time period information)

[1523] output

[1524] Search results (corresponding past image data)

[1525] Specific actions

[1526] The server executes database queries based on the specified latitude, longitude, and date ranges and collects the corresponding image data.

[1527] Step 3:

[1528] The server analyzes the image data and extracts the metadata.

[1529] The server analyzes the acquired image data using AI image recognition technology and extracts metadata for each image, including the date and location of the image, as well as information about related events.

[1530] input

[1531] Search results (past image data)

[1532] output

[1533] Image data with metadata

[1534] Specific actions

[1535] The server uses an AI model (e.g., an image recognition model) to analyze image data and automatically extract metadata such as the date and location of the photograph, and information about the event.

[1536] Step 4:

[1537] The server adds location information and stores it in the database.

[1538] The server adds location information (latitude and longitude) to each image data based on the extracted metadata and stores it in the database as image data with location information.

[1539] input

[1540] Image data with metadata

[1541] output

[1542] Image data with location information

[1543] Specific actions

[1544] The server calculates the latitude and longitude information corresponding to each image based on the metadata, adds location information, and stores it in the database.

[1545] Step 5:

[1546] The server integrates current map data and historical image data and transmits it.

[1547] The server integrates location-based image data with current map data, converts it into a data format that allows the user to compare past images with the current map, and sends it to the terminal.

[1548] input

[1549] Image data with location information

[1550] Current map data

[1551] output

[1552] Integrated data (data from past images and current maps)

[1553] Specific actions

[1554] The server overlays historical images onto map data based on location information and converts them into a format that allows for comparison and display of both.

[1555] Step 6:

[1556] The device displays integrated data and provides explanatory text.

[1557] The device displays the received integrated data and compares past and present images. It also displays explanatory text related to the images.

[1558] input

[1559] Integrated data (data from past images and current maps)

[1560] output

[1561] Images and descriptions displayed on the user interface

[1562] Specific actions

[1563] The device displays integrated data in a user-friendly format and provides a convenient interface for comparing past and present situations.

[1564] Step 7:

[1565] The server provides relevant product information based on the user's interests.

[1566] The server generates relevant product information based on past image data, descriptions, and user operation data, and transmits this information to the terminal.

[1567] input

[1568] User operation data

[1569] Past image data

[1570] Description

[1571] output

[1572] Related product information

[1573] Specific actions

[1574] The server uses a generative AI model to analyze user interests and generate data to select and provide corresponding product information.

[1575] Step 8:

[1576] The device displays related product information and enables the purchase process.

[1577] The device displays the received related product information, allowing the user to purchase items they are interested in directly.

[1578] input

[1579] Related product information

[1580] output

[1581] Product information and purchase options displayed on the user interface

[1582] Specific actions

[1583] The terminal displays related product information and provides an interface that allows users to proceed directly to the purchase process by clicking on a product.

[1584] The above outlines the specific processing steps. This processing flow allows users to easily compare past and current situations and gain an effective experience in purchasing related products.

[1585] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[1586] This invention combines a system that collects historical image data, organizes it based on metadata, stores it in a database with location information, searches it based on the user's specified location, and displays the integrated historical image data and current map data with an emotion engine that recognizes the user's emotions. This enables the provision of optimal information according to the user's emotional state.

[1587] System Configuration

[1588] 1. Server: This server is central to the present invention, performing data processing and storage, and responding to user requests. It also incorporates an emotion engine to analyze user emotions.

[1589] 2. Device: A device (such as a smartphone, tablet, or PC) that a user uses to access and operate the system through an application or website.

[1590] 3. User: A person who specifies a location, views the results, and performs operations through a dedicated interface.

[1591] System operation

[1592] First, historical image data is collected. The server executes a script that automatically retrieves image data from publicly available databases on the internet. The collected image data is temporarily stored in storage.

[1593] Next, the server analyzes the collected image data using AI image recognition technology. The AI ​​extracts metadata from the images, including the date, location, and events that occurred. Based on this metadata, the server adds location information (latitude and longitude) to each image.

[1594] The image data with added location information is stored in a specialized database. This database is indexed later to enable efficient searching.

[1595] Users access the system through a dedicated application or website. Users specify a location they want to view on a map and submit a request. The server receives this request and searches its database for historical image data corresponding to the specified location.

[1596] The server organizes the search results and prepares image datasets related to the specified locations. It filters out any sensitive information to ensure that the images are suitable for publication.

[1597] This is where the emotion engine comes in. The emotion engine analyzes the user's facial expressions and voice to recognize the user's emotions in real time. Emotions are categorized into positive, negative, and neutral.

[1598] The server optimizes the information it provides based on the user's emotional state. For example, if a user is showing negative emotions, it prioritizes providing image data with generally positive content. It also performs filtering corresponding to specific emotional states to prevent the display of information that may be offensive to the user.

[1599] Finally, the server sends the filtered data to the user's terminal. The terminal analyzes the received data and displays the historical image data and current map data integrated on the interface. The user can simultaneously view historical and current information for a specified location.

[1600] Specific example

[1601] For example, consider a scenario where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. Meanwhile, the emotion engine recognizes "positive emotions" from the user's facial expressions and voice. This information is sent to the server.

[1602] The server searches the database for entries containing historical image data related to a specified location in Shinjuku Ward, retrieving entries such as "a photograph of a residential area in Shinjuku Ward in the 1930s" or "positive anecdotes from the reconstruction period after the Great Kanto Earthquake." Simultaneously, the sentiment engine optimizes the data to provide an overall positive content, taking into account the user's positive emotions.

[1603] The server filters this data, excluding sensitive information, and then sends the organized data to the user's device. The device receives the data and displays past images and their descriptions on the screen along with a current map. Users can easily compare the past and present while prioritizing the viewing of positive information.

[1604] Thus, the present invention is a system that efficiently links past image data and current map data, and further utilizes an emotion engine to provide optimal information tailored to the user's emotional state.

[1605] The following describes the processing flow.

[1606] Step 1:

[1607] Server: Executes scripts to access publicly available databases and libraries on the internet and collect historical image data. The collected image data is stored in temporary storage.

[1608] Step 2:

[1609] Server: The server initiates analysis of the collected image data using AI image recognition technology. The AI ​​model extracts metadata from the images, including the date, location, and events that occurred.

[1610] Step 3:

[1611] Server: Based on the extracted metadata, it adds location information such as latitude and longitude to each image. This information is used to set location attributes for the image data.

[1612] Step 4:

[1613] Server: Stores location-tagged image data in a database. An index is created during storage to enable efficient searching later.

[1614] Step 5:

[1615] User: Access the dedicated application or website and log in. The user uses the map interface to specify the location for which they want to view past information by clicking or tapping.

[1616] Step 6:

[1617] Terminal: Sends the user's location information as a request to the server. The request includes the latitude and longitude information of the specified location.

[1618] Step 7:

[1619] Server: Based on the received location request, it searches the database. It retrieves historical image data and its metadata associated with the specified latitude and longitude.

[1620] Step 8:

[1621] Server: Organizes search results and prepares image datasets related to the specified locations. Filters out sensitive information to ensure it is appropriate for publication.

[1622] Step 9:

[1623] Server: Sends filtered data to the user's terminal. The data is sent in a structured format such as JSON.

[1624] Step 10:

[1625] Terminal: Analyzes received data and integrates historical image data and current map data on the interface for display. Users can simultaneously view historical and current information for a specified location.

[1626] Step 11:

[1627] User: Uses the device's camera and microphone to express emotions through facial expressions and voice. The emotion engine acquires this data and analyzes emotions in real time.

[1628] Step 12:

[1629] Server: The emotion engine classifies the user's emotions as positive, negative, or neutral, and provides optimized information based on that emotional state.

[1630] Step 13:

[1631] Server: Filters relevant metadata based on emotional state. For example, if a user is expressing negative emotions, it prioritizes providing image data with positive content.

[1632] Step 14:

[1633] Server: Retransmits the information set optimized by the emotion engine to the user's terminal.

[1634] Step 15:

[1635] Device: Redisplays past images and their descriptions that match the user's emotional state, along with the current map. Users can enjoy viewing information in an emotionally sensitive way.

[1636] This series of steps allows users to view historical image data and current map data in an integrated manner, and then receive optimized information from the emotion engine.

[1637] (Example 2)

[1638] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1639] Conventional systems that collect historical image data, add location information, store it in a database, and search based on the user's specified location do not consider the user's emotional state, and the information provided may not always be optimal. Furthermore, if sensitive information is not filtered, inappropriate information may be provided. To solve this problem, a system is needed that provides optimal information tailored to the user's emotional state.

[1640] The identification processing performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting past image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the image data with location information in a database, means for receiving a location specification from the user, means for searching the database based on the location specification, means for providing past image data as a search result, means for integrating and displaying past image data and current map data, and an emotion engine that recognizes the user's emotions, analyzes the user's emotions using the emotion engine, and optimizes the information to be provided based on the analysis results. This makes it possible to provide optimal information according to the user's emotional state.

[1641] "Past image data" refers to photographs and image files taken in the past, including data such as the date, time, and location of the shooting.

[1642] "Metadata" refers to supplementary information related to image data, including data such as the date and time of shooting, location, and details of the event.

[1643] "Location information" refers to longitude and latitude data used to indicate a specific location.

[1644] A "database" refers to a specialized data storage system for efficiently storing, searching, and managing image data, its metadata, and location information.

[1645] A "user" refers to a person who specifies a location or performs other operations through the system.

[1646] "Location specification" refers to the user selecting a location they want to see on a map or specifying it in text.

[1647] An "emotion engine" refers to a system equipped with technology that analyzes a user's emotions in real time from their facial expressions and voice, and classifies that emotional state as positive, negative, neutral, etc.

[1648] "Integration" refers to combining historical image data and current map data and displaying them on a single interface.

[1649] "Optimization" refers to adjusting the content and order of information provided based on the user's emotional state, as analyzed by the emotion engine, to make the user experience as effective as possible.

[1650] "Sensitive information" refers to information that should be restricted from public disclosure based on specific criteria, such as personal information or data deemed socially inappropriate.

[1651] Modes for carrying out the invention

[1652] This invention combines a system that collects historical image data, organizes it based on metadata, stores it in a database with location information, searches it based on the user's specified location, and displays the integrated historical image data and current map data with an emotion engine that recognizes the user's emotions. This enables the provision of optimal information according to the user's emotional state.

[1653] Hardware and software configuration

[1654] 1. Server:

[1655] The server performs the central data processing and storage of this invention, as well as responding to user requests. It also incorporates an emotion engine to analyze user emotions. This requires a high-performance server, utilizing software such as Python, MySQL, and OpenCV.

[1656] 2. Terminal:

[1657] A terminal is a device that a user uses to access and operate a system through an application or website. Typically, smartphones, tablets, and PCs are used.

[1658] 3. User:

[1659] A user is a person who specifies a location, views the results, and performs actions through a dedicated interface.

[1660] Program Processing Description

[1661] First, historical image data is collected. The server executes a script to automatically retrieve image data from publicly available databases on the internet. The collected image data is temporarily stored in the server's storage. Next, the server analyzes the collected image data using AI image recognition technology (e.g., OpenCV models). The AI ​​extracts metadata from the images, such as the date and location of the image, and events, and adds location information (latitude and longitude) based on this metadata. The image data with location information is stored in a specialized database, such as a MySQL database, and indexed to enable efficient searching.

[1662] Users access the system through a dedicated application or website. Users specify a location they wish to view on a map and submit a request. The server receives this request and searches its database for historical image data corresponding to the specified location. The server organizes the search results and prepares an image dataset related to the specified location. During this process, it filters out sensitive information and verifies whether the data is appropriate for publication.

[1663] The addition of an emotion engine allows the system to analyze the user's facial expressions and voice to recognize their emotions in real time. Emotions are categorized as positive, negative, and neutral, and the server optimizes the information it provides based on the user's emotional state. For example, if a user is showing negative emotions, the system prioritizes providing image data with generally positive content. It also filters content based on specific emotional states to prevent the display of information that may be unpleasant to the user.

[1664] Finally, the server sends the filtered data to the user's terminal. The terminal analyzes the received data and can integrate and display historical image data and current map data on its interface. The user can simultaneously view historical and current information for a specified location.

[1665] Specific example

[1666] For example, consider a scenario where a user specifies a residential area in Shinjuku Ward. The user launches the application and selects a location in Shinjuku Ward on the map. Meanwhile, the emotion engine recognizes "positive emotions" from the user's facial expressions and voice. This information is sent to the server. The server searches its database for entries containing historical image data related to the specified location in Shinjuku Ward, retrieving, for example, "a photograph of a residential area in Shinjuku Ward in the 1930s" or "positive episodes from the reconstruction period after the Great Kanto Earthquake." At the same time, the emotion engine optimizes the data to provide overall positive content, taking into account the user's positive emotions. The server filters this data, excluding sensitive information, and then sends the organized data to the user's device. The device receives the data and displays the historical images and their descriptions on the screen along with the current map. The user can easily compare the past and present while prioritizing the viewing of positive information.

[1667] Thus, the present invention is a system that efficiently links past image data and current map data, and further utilizes an emotion engine to provide optimal information tailored to the user's emotional state.

[1668] Example of a prompt

[1669] The following are examples of prompt statements to input into the generating AI model.

[1670] Example 1:

[1671] Search for image data of "Shinjuku Ward, 1930s residential area" and provide information that includes positive episodes. The user's emotional state is positive.

[1672] Example 2:

[1673] Display image data related to the "episode from the reconstruction period after the Great Kanto Earthquake" specified by the user. The emotion engine will analyze the user's emotional state, and if it indicates positive emotions, prioritize providing data with positive content.

[1674] The flow of the specific processing in Example 2 will be explained using Figure 13.

[1675] Step 1:

[1676] Image data collection:

[1677] The server executes a script to automatically retrieve image data from a publicly available database on the internet. Specifically, it uses the Python requests library to send an HTTP request from a specific URL and download the image data. The input for this step is the URL of the public database, and the output is the downloaded image file.

[1678] Specific examples of operation:

[1679] The server accesses https: / / example.com / api / images and downloads the image data.

[1680] Step 2:

[1681] Image data analysis:

[1682] The server analyzes the collected image data using AI image recognition technology (e.g., OpenCV models). The input is the downloaded image data, and the output is metadata about the date, location, and event of the photograph. The AI ​​extracts this metadata from text information and digital prints within the image.

[1683] Specific examples of operation:

[1684] The server uses OpenCV and pytesseract to analyze image data and extract text information.

[1685] Step 3:

[1686] Adding location information:

[1687] The server uses the extracted metadata to add location information (latitude and longitude) to the image data. The input is metadata about the shooting location, and the output is location-attached image data containing latitude and longitude information. In this step, the metadata is sent to a geocoding API to obtain the latitude and longitude.

[1688] Specific examples of operation:

[1689] The server sends location information for the filming location in Shinjuku Ward to a geocoding API to obtain the latitude and longitude.

[1690] Step 4:

[1691] Saving to the database:

[1692] The server stores location-tagged image data in a specialized database. The input is location-tagged image data, and the output is the data entries stored in the database. For storage efficiency, the server stores the image data itself, its links, metadata, location information, etc.

[1693] Specific examples of operation:

[1694] The server connects to a MySQL database and stores image data, including location information and metadata.

[1695] Step 5:

[1696] Receiving user requests:

[1697] Users operating the terminal access the system through a dedicated application or website, specify the location they want to view on a map, and send a request. The input is the location specified by the user, and the output is a search request related to that location.

[1698] Specific examples of operation:

[1699] The user launches the application, clicks on a location in Shinjuku Ward on the map, and sends a request.

[1700] Step 6:

[1701] Searching and filtering data:

[1702] The server searches its database for historical image data corresponding to a specified location based on the received user request. The input is the user's request, and the output is the associated image dataset. Simultaneously, sensitive information is filtered.

[1703] Specific examples of operation:

[1704] The server searches the database for entries related to Shinjuku Ward, filters out sensitive information, and retrieves the results.

[1705] Step 7:

[1706] Emotion recognition by an emotion engine:

[1707] The user's facial expressions and voice are collected through the device's camera and microphone, and the server's emotion engine analyzes them in real time. The input is the user's facial expressions and voice data, and the output is categorized as positive, negative, or neutral emotions.

[1708] Specific examples of operation:

[1709] The server uses EmotionEngine to analyze the user's real-time data and recognizes their emotional state as positive.

[1710] Step 8:

[1711] Optimizing the information provided:

[1712] The server optimizes the information it provides based on the user's emotional state. The input consists of emotion recognition results and historical image data, while the output is an optimized image dataset. For example, if a negative emotion is indicated, positive image data is prioritized.

[1713] Specific examples of operation:

[1714] The server considers the emotional state to be positive and filters out data that includes positive episodes.

[1715] Step 9:

[1716] Sending and displaying data:

[1717] The server sends optimized data to the user's terminal. The input is optimized image data, and the output is data transmission to the terminal. The terminal analyzes the received data and displays it on the interface, integrating historical image data and current map data. This allows the user to view historical and current information for a specified location simultaneously.

[1718] Specific examples of operation:

[1719] The server sends optimized image data to the user's terminal, which then displays it on the screen along with the current map data.

[1720] (Application Example 2)

[1721] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[1722] This invention relates to a system that collects and organizes past image data and searches and displays it based on a location specified by the user. However, conventional systems provide uniform information without considering the user's emotional state, resulting in the inability to provide information tailored to the individual user's emotions and needs. As a result, the user experience may be compromised. This invention aims to solve this problem and realize the provision of optimal information according to the user's emotional state.

[1723] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[1724] In this invention, the server includes means for collecting past image data, means for extracting metadata from the image data, means for adding location information based on the metadata, means for storing the location-attached image data in a database, means for receiving a location specification from a user, means for searching the database based on the location specification, means for providing past image data as a search result, means for integrating and displaying the past image data and current map data, and means for analyzing the user's emotions and optimizing the image data provided based on those emotions. This makes it possible to provide information that takes into account the user's emotional state.

[1725] "Past image data" refers to digital data of photographs and videos taken in the past.

[1726] "Metadata" refers to information associated with image data, including details about the date and location of the photograph, and events that occurred.

[1727] "Location information" refers to data that indicates a geographical location, and is usually represented by latitude and longitude.

[1728] A "database" is a system designed to efficiently store and retrieve large amounts of data.

[1729] "Location specification" refers to the operation in which a user selects or specifies a particular geographical location.

[1730] "Map data" refers to digital data containing geographical information, showing current topography and the locations of buildings.

[1731] "Emotion analysis" is a technology that identifies emotions from a user's facial expressions and voice, and analyzes their emotional state.

[1732] "Searching" is the process of exploring a database based on specified conditions and extracting relevant data.

[1733] "Integrated display" is the operation of displaying multiple data points together on a single screen.

[1734] "Optimization" means adjusting or improving something to best suit a specific purpose.

[1735] This invention combines a system that collects historical image data, organizes it based on metadata, stores it in a database with location information, searches it based on the user's specified location, and displays the integrated historical image data and current map data with an emotion engine that recognizes the user's emotions. This enables the provision of optimal information according to the user's emotional state.

[1736] The system's core is the server. The server handles data processing, storage, and responding to user requests. It also incorporates an emotion recognition engine to analyze user emotions. Specifically, the server includes the following:

[1737] 1. Collection of image data

[1738] The server executes a script to automatically retrieve image data from publicly available databases on the internet. This temporarily stores past image data in storage.

[1739] 2. Extraction of metadata

[1740] The server analyzes the collected image data using AI image recognition technology (specifically, TensorFlow and Keras can be used). The AI ​​extracts metadata from the images, including the date, location, and events that occurred. Based on this metadata, the server adds location information (latitude and longitude) to each image.

[1741] 3. Storing and retrieving data in the database

[1742] The location-tagged image data is stored in a specialized database (MongoDB or PostgreSQL are examples). This database is indexed to enable efficient searching later. Based on the location specified by the user, the server searches the database and retrieves historical image data corresponding to the specified location.

[1743] 4. Emotion Recognition and Optimization

[1744] An emotion recognition engine (specifically, Microsoft Azure's Emotion API or Google Cloud AI's Vision AI can be used) analyzes the user's facial expressions and voice to recognize their emotions in real time. Emotions are categorized as positive, negative, and neutral. The server optimizes the information it provides based on the user's emotional state. For example, it prioritizes providing positive image data to a user exhibiting negative emotions.

[1745] 5. Integrated display

[1746] The server sends filtered data to the user's device (smart glasses, smartphone, etc.). The device analyzes the received data and displays it by integrating historical image data and current map data on its interface. For example, map data can be displayed using the Google Maps API or Apple Maps.

[1747] Specific example

[1748] For example, consider a scenario where a user specifies a tourist spot in Shinjuku Ward. The user launches a dedicated application and selects a specific location in Shinjuku Ward on a map. Simultaneously, the user's facial expressions and voice are analyzed via camera and microphone, and an emotion engine recognizes "positive emotions." This information is then sent to the server.

[1749] The server searches its database for historical image data related to a specified location in Shinjuku Ward (e.g., photos of Shinjuku Ward from the 1930s or positive anecdotes) and provides it. The sentiment engine prioritizes selecting data with an overall positive content, taking into account that the user has expressed positive emotions.

[1750] The server filters this data, excluding sensitive information, and then sends the organized data to the user's device. The user's device receives the data and displays past images and their descriptions on the screen along with the current map. In this way, the user can compare the past and present while prioritizing the viewing of positive information.

[1751] Example of a prompt

[1752] "Integrate historical image data with current map data and provide information that matches the user's positive emotions. For example, if a user is moved, display historically positive photos or stories about that place."

[1753] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[1754] Step 1:

[1755] Users access the system through smart glasses or a smartphone app and specify a particular geographical location. This input includes information about the geographical location specified by the user (latitude and longitude).

[1756] Specific operation: The user launches a map application and taps or clicks on a specific location in Shinjuku Ward on the map to specify it.

[1757] Step 2:

[1758] The server receives location information sent by the user and searches the database for historical image data associated with that location. The input here is the geographical location information specified by the user, and the output is the corresponding historical image data.

[1759] Specific operation: The server generates a database query to search for image data corresponding to the specified latitude and longitude. MongoDB or PostgreSQL can be used as the database.

[1760] Step 3:

[1761] The server analyzes the collected image data using AI image recognition technology and extracts metadata (date of capture, location, and event). The input here is historical image data, and the output is the associated metadata.

[1762] Specific operation: As AI image recognition technology, TensorFlow and Keras are used to extract necessary information from image data.

[1763] Step 4:

[1764] The server analyzes the user's emotions using data acquired from the user's smart glasses or smartphone camera / microphone. The input is the user's facial expression images and audio data, and the output is identified emotion information.

[1765] Specific operation: An emotion recognition engine (e.g., Microsoft Azure's Emotion API) is used to analyze the user's facial expressions and voice to determine the category of emotion (positive, negative, neutral).

[1766] Step 5:

[1767] The server selects the most suitable image data based on the analyzed sentiment information. The input here is image data with sentiment information and metadata, and the output is historical image data optimized for the user's sentiment.

[1768] Specific operation: The server refers to emotional information and, for example, prioritizes selecting bright and hopeful images for users who exhibit positive emotions.

[1769] Step 6:

[1770] The server sends image data optimized according to the user's emotions to the user's device. Here, the input is the optimized image data, and the output is the content displayed on the user's device.

[1771] Specific operation: The server organizes the selected image data, bundles it into a format such as JSON, and sends it to the user's smart glasses or smartphone application.

[1772] Step 7:

[1773] The terminal integrates and displays historical image data and current map data based on the received data. The input consists of image data and map data sent from the server, and the output is an integrated display screen that the user can view.

[1774] Specific operation: The device will use the Google Maps API or Apple Maps to build an interface that overlays current map data with historical image data.

[1775] Through the above processing steps, users can simultaneously compare the past and present of their location while receiving information optimized for their emotions at that time.

[1776] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[1777] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1778] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[1779] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1780] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[1781] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[1782] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[1783] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[1784] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[1785] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[1786] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[1787] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[1788] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[1789] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1790] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[1791] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[1792] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[1793] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[1794] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[1795] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[1796] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[1797] The following is further disclosed regarding the embodiments described above.

[1798] (Claim 1)

[1799] Means for collecting past image data,

[1800] Means for extracting metadata from the aforementioned image data,

[1801] Means for adding location information based on the aforementioned metadata,

[1802] means for storing the aforementioned image data with location information in a database,

[1803] A means of receiving location information from the user,

[1804] Means for searching the database based on the location specified above,

[1805] A means of providing past image data as search results,

[1806] A system including means for integrating and displaying the aforementioned past image data and current map data.

[1807] (Claim 2)

[1808] The system according to claim 1, characterized in that the means for collecting the image data includes means for automatically acquiring image data from a publicly available database on the Internet.

[1809] (Claim 3)

[1810] The system according to claim 1, characterized in that the metadata extraction means includes means for analyzing information regarding the date, location, and event of the image data using artificial intelligence technology.

[1811] (Claim 4)

[1812] The system according to claim 1, characterized in that the means for storing the location-information-attached image data in a database includes means for creating an index to improve search efficiency.

[1813] (Claim 5)

[1814] The system according to claim 1, characterized in that it includes means for filtering out sensitive information from past image data provided as a search result.

[1815] "Example 1"

[1816] (Claim 1)

[1817] Means for collecting past image data,

[1818] Means for extracting metadata from the aforementioned image data,

[1819] Means for adding location information based on the aforementioned metadata,

[1820] means for storing the aforementioned image data with location information in a database,

[1821] A means of receiving location information from the user,

[1822] Means for searching the database based on the location specified above,

[1823] A means of providing past image data as search results,

[1824] A means for integrating and displaying the aforementioned past image data and current map data,

[1825] Methods for filtering search results,

[1826] Means for performing indexing,

[1827] A system that includes this.

[1828] (Claim 2)

[1829] The system according to claim 1, characterized in that the means for collecting the image data includes means for automatically acquiring image data from a publicly available database on the Internet.

[1830] (Claim 3)

[1831] The system according to claim 1, characterized in that the metadata extraction means includes means for analyzing information regarding the date, location, and event of the image data using artificial intelligence technology.

[1832] "Application Example 1"

[1833] (Claim 1)

[1834] Means for collecting past image data,

[1835] Means for extracting metadata from the aforementioned image data,

[1836] Means for adding location information based on the aforementioned metadata,

[1837] means for storing the aforementioned image data with location information in a database,

[1838] A means of receiving location information from the user,

[1839] Means for searching the database based on the location specified above,

[1840] A means of providing past image data as search results,

[1841] A means for integrating and displaying the aforementioned past image data and current map data,

[1842] A means of displaying past and present image data and providing explanatory text,

[1843] A means of providing relevant product information based on user interests and enabling purchases.

[1844] A system that includes this.

[1845] (Claim 2)

[1846] The system according to claim 1, characterized in that the means for collecting the image data includes means for automatically acquiring image data from a publicly available database on the Internet.

[1847] (Claim 3)

[1848] The system according to claim 1, characterized in that the metadata extraction means includes means for analyzing information regarding the date, location, and event of the image data using artificial intelligence technology.

[1849] "Example 2 of combining an emotion engine"

[1850] (Claim 1)

[1851] Means for collecting past image data,

[1852] Means for extracting metadata from the aforementioned image data,

[1853] Means for adding location information based on the aforementioned metadata,

[1854] means for storing the aforementioned image data with location information in a database,

[1855] A means of receiving location information from the user,

[1856] Means for searching the database based on the location specified above,

[1857] A means of providing past image data as search results,

[1858] A means for integrating and displaying the aforementioned past image data and current map data,

[1859] Includes an emotion engine that recognizes the user's emotions,

[1860] A system that includes means for analyzing the user's emotions using the aforementioned emotion engine and optimizing the information provided based on the analysis results.

[1861] (Claim 2)

[1862] The system according to claim 1, characterized in that the means for collecting the image data includes means for automatically acquiring image data from a publicly available database on the Internet.

[1863] (Claim 3)

[1864] The system according to claim 1, characterized in that the metadata extraction means includes means for analyzing information regarding the date, location, and event of the image data using artificial intelligence technology.

[1865] "Application example 2 when combining with an emotional engine"

[1866] (Claim 1)

[1867] Means for collecting past image data,

[1868] Means for extracting metadata from the aforementioned image data,

[1869] Means for adding location information based on the aforementioned metadata,

[1870] means for storing the aforementioned image data with location information in a database,

[1871] A means of receiving location information from the user,

[1872] Means for searching the database based on the location specified above,

[1873] A means of providing past image data as search results,

[1874] A means for integrating and displaying the aforementioned past image data and current map data,

[1875] A system including means for analyzing a user's emotions and optimizing the image data provided based on those emotions.

[1876] (Claim 2)

[1877] The system according to claim 1, characterized in that the means for collecting the image data includes means for automatica...

Claims

1. Means for collecting past image data, Means for extracting metadata from the aforementioned image data, Means for adding location information based on the aforementioned metadata, means for storing the aforementioned image data with location information in a database, A means of receiving location information from the user, Means for searching the database based on the location specified above, A means of providing past image data as search results, A system including means for integrating and displaying the aforementioned past image data and current map data.

2. The system according to claim 1, characterized in that the means for collecting the image data includes means for automatically acquiring image data from a publicly available database on the Internet.

3. The system according to claim 1, characterized in that the metadata extraction means includes means for analyzing information regarding the date, location, and event of the image data using artificial intelligence technology.

4. The system according to claim 1, characterized in that the means for storing the location-information-attached image data in a database includes means for creating an index to improve search efficiency.

5. The system according to claim 1, characterized in that it includes means for filtering out sensitive information from past image data provided as a search result.

Images