System

Abstract

To provide a system that reproduces past townscapes and enables a user to experience them in conjunction with the real world.SOLUTION: A system comprises: means for acquiring, based on a user's location information, an old map being the past map relative to the present and relating to the periphery of the location information and an old photograph being the past photograph relative to the present and relating to the periphery of the location information; means for generating a three-dimensional model on the basis of a prompt sentence for instructing generation of the three-dimensional model representing a landscape being the past landscape relative to the present based on the old map and the old photograph and relating to the periphery of the location information, and a generative AI model; and means for displaying an image obtained on the basis of the generated three-dimensional model on smart glasses worn by the user.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] With current technology, it is difficult to use information from old maps and photographs to recreate cityscapes of the past and experience them in conjunction with the real world. [Means for solving the problem]

[0005] This invention works in conjunction with smart glasses to analyze information from old maps and old photographs. It generates old streetscapes based on the analyzed information and displays the generated streetscapes on the smart glasses. This allows users to have the experience of traveling back in time. [Brief description of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a process flow of the data processing system in the first embodiment of the first form example. [Figure 12] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the application example 1 of the first embodiment. [Figure 13] FIG. 11 is a sequence diagram showing a process flow of the data processing system in the second embodiment of the second embodiment. [Figure 14] FIG. 13 is a sequence diagram showing a process flow of the data processing system in application example 2 of embodiment 2. [Figure 15] FIG. 13 is a sequence diagram showing a process flow of a data processing system in a third embodiment of the present invention. [Figure 16] FIG. 13 is a sequence diagram showing a process flow of the data processing system in the third application example of the third embodiment. [Figure 17]FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 13 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs voice in accordance with instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0021] It's a camera.

[0022] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0023] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0024] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0025] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0026] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0027] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0028] "Example 1"

[0029] As one embodiment of the present invention, a system that works with smart glasses is provided. The system includes a module that analyzes information on old maps and old photographs. The analysis module extracts geographical features and architectural features from the old maps and old photographs. A module that generates old streetscapes operates based on the extracted features. The generation module generates a 3D model based on the extracted geographical features and architectural features. The generated streetscape is displayed on the smart glasses, allowing the user to experience a time-traveling experience.

[0030] "Example 2"

[0031] For example, when a user wears smart glasses and stands at a specific location, information about old maps and old photographs of that location is sent to the system. The system analyzes the information and generates an old streetscape of that location. The generated streetscape is displayed on the smart glasses, allowing the user to experience a fusion of the real world and the old streetscape.

[0032] The process flow of each embodiment will be described below.

[0033] "Example 1"

[0034] Step 1: A user puts on the smart glasses and stands at a specific point.

[0035] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[0036] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[0037] Step 4: The module that generates old streetscapes runs based on the extracted features.

[0038] Step 5: The generated streetscape is displayed on the smart glasses, giving the user an experience that blends the real world with the old streetscape.

[0039] "Example 2"

[0040] Step 1: A user puts on the smart glasses and stands at a specific point.

[0041] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[0042] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[0043] Step 4: The module that generates old streetscapes runs based on the extracted features.

[0044] Step 5: The generated streetscape is displayed on the smart glasses, giving the user an experience that blends the real world with the old streetscape.

[0045] Example 1

[0046] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0047] With conventional technology, it was difficult to reproduce past streetscapes using old maps and old photographs. In addition, the means by which users could experience past streetscapes were limited. This restricted the use of the technology in fields such as the reproduction of historical streetscapes, education, and tourism.

[0048] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0049] In this invention, the server includes a means for a user to upload old maps and old photographs from a terminal, a means for the server to receive the uploaded data, a means for the server to analyze the data using an analysis module and extract geographical features and architectural features, a means for the server to generate a 3D model using a generation module based on the features extracted by the server, a means for the server to transmit the generated 3D model to the smart glasses, and a means for the smart glasses to display the 3D model and for the user to experience it. This allows the user to experience past streetscapes in real time.

[0050] A "user" is an individual or group that uses the system to upload old maps and old photographs and experience streetscapes of the past.

[0051] A "terminal" is a device that a user uses to upload old maps and old photographs, and includes smartphones, personal computers, etc.

[0052] "Server" refers to a computer system that receives data uploaded by users and performs analysis and generation of 3D models.

[0053] The "analysis module" is a software component that analyzes data from old maps and old photographs and extracts geographical and architectural features.

[0054] "Geographical features" are geographical elements such as rivers, roads, and mountains that are included in old maps and old photographs.

[0055] "Building characteristics" refers to information such as the shape and layout of buildings contained in old maps and photographs.

[0056] The "generative module" is a software component for generating 3D models based on extracted geographical and architectural features.

[0057] A "3D model" is a three-dimensional digital representation generated based on extracted features.

[0058] "Smart glasses" are a wearable device that allows users to visually experience three-dimensional models.

[0059] "Uploading" refers to the act of a user using a terminal to send data of old maps or old photographs to a server.

[0060] "Receiving" is the act of the server taking in data sent by the user.

[0061] "Transmitting" refers to the act of the server sending the generated 3D model to the smart glasses.

[0062] "Display" refers to the act of the smart glasses projecting the received 3D model into the user's field of vision.

[0063] An "experience" is an act in which a user can visually sense a streetscape of the past through the smart glasses.

[0064] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0065] This invention is a system that allows users to experience past streetscapes using old maps and old photographs. This system includes a series of processes in which the user uploads old maps and old photographs from a terminal, the server analyzes the data, generates a 3D model, and displays it to the user through smart glasses.

[0066] Hardware and Software Used

[0067] Terminal

[0068] Users upload old maps and old photographs using devices such as smartphones and computers, allowing them to easily access the system and provide data.

[0069] server

[0070] The server receives data uploaded by users, analyzes it, and generates 3D models. The following software is installed on the server:

[0071] Analysis module: Uses machine learning frameworks such as Python's OpenCV library and TENSORFLOW(R) to analyze data from old maps and photographs and extract geographic and architectural features.

[0072] Generation module: Using 3D modeling software such as Blender or Unity, a 3D model is generated based on the extracted features.

[0073] Smart Glasses

[0074] The smart glasses display the 3D model sent from the server in the user's field of vision, allowing the user to experience past streetscapes in real time.

[0075] Examples

[0076] For example, if a user uploads an old map of Tokyo in the Edo period in JPEG format, the server receives this data and uses OpenCV to detect the outlines of rivers, roads, and buildings.The server then uses Blender to generate a 3D model of the streets of Tokyo in the Edo period based on the detected features.This 3D model is sent to the smart glasses via Wi-Fi, allowing the user to experience Tokyo in the Edo period through the smart glasses.

[0077] Examples of prompt statements

[0078] Examples of prompts to be input to a generative AI model include the following:

[0079] "Analyze old maps of Tokyo from the Edo period and generate a 3D model of the cityscape from that time."

[0080] Using this prompt, the generative AI model can perform the necessary analysis and generation processes to provide the user with an experience that recreates a historical cityscape.

[0081] In this way, the present invention provides a system that allows a user to experience past streetscapes in real time.

[0082] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0083] Program processing flow

[0084] Step 1: The user uploads old maps and old photos from their device.

[0085] Users upload data on old maps and old photographs to the system using devices such as smartphones or computers. For example, an old map of Tokyo from the Edo period can be uploaded in JPEG format.

[0086] Input: Data of old maps and old photographs (e.g. JPEG format images)

[0087] Output: Uploaded data is sent to the server

[0088] Step 2: The server receives the uploaded data

[0089] The server receives data of old maps and old photographs uploaded by users. This data is stored in a specific directory on the server.

[0090] Input: Data on old maps and old photographs sent by users

[0091] Output: Data stored in a specific directory on the server

[0092] Step 3: The server analyzes the data using the analysis module to extract geographic and architectural features.

[0093] The server uses machine learning frameworks such as Python's OpenCV library and TensorFlow to analyze the uploaded data, for example to detect the outlines of rivers, roads, and buildings in old maps and extract their features.

[0094] Input: Data of old maps and old photographs stored on the server

[0095] Output: Extracted geographic and architectural features (e.g. rivers, roads, building outlines)

[0096] Step 4: Based on the features extracted by the server, a 3D model is generated using the generation module.

[0097] The server uses 3D modeling software such as Blender or Unity to generate 3D models based on the extracted features, for example recreating the streets of Edo-period Tokyo based on the location information of detected rivers and roads.

[0098] Input: Extracted geographic and building features

[0099] Output: Generated 3D model

[0100] Step 5: The server sends the generated 3D model to the smart glasses

[0101] The server then transmits the generated 3D model to the smart glasses via wireless communication such as Wi-Fi or Bluetooth.

[0102] Input: Generated 3D model

[0103] Output: 3D model sent to smart glasses

[0104] Step 6: The smart glasses display the 3D model and the user experiences it.

[0105] The smart glasses then display the received 3D model in the user's field of vision, giving the user the experience of being transported back in time to Edo-period Tokyo.

[0106] Input: 3D model sent to smart glasses

[0107] Output: A 3D model displayed in the user's field of view.

[0108] In this way, the system allows users to experience past streetscapes in real time.

[0109] (Application example 1)

[0110] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0111] In modern brick-and-mortar stores, it is difficult for users to experience the streetscapes and buildings of the past. This limits opportunities to understand historical background and culture. In addition, there is a lack of ways to visually reproduce past scenery in tourist spots and historical places, making it difficult to attract users' interest.

[0112] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscape on the smart glasses, and a means for allowing the user to experience past streetscapes in a real store. This allows the user to visually experience the appearance of past streetscapes and buildings in a real store, and to gain a deeper understanding of the historical background and culture.

[0113] "Smart glasses" are glasses-type devices that users can wear to provide augmented reality and display information.

[0114] An "antique map" is a map that shows geographical information from the past, recording historical geographical features and the layout of buildings.

[0115] "Old photographs" are photographs of landscapes, buildings, and people from the past, providing visual historical information.

[0116] "Means of analysis" refers to technologies and devices for digitally processing information from old maps and old photographs and extracting geographical and architectural features.

[0117] "Means of generation" refers to technology and equipment for creating 3D models of past cityscapes and buildings based on analyzed information.

[0118] "Display means" refers to the technology or devices for displaying the generated 3D model on the smart glasses and providing the user with a visual experience.

[0119] A "brick and mortar store" is a store that exists in a physical location and where users can visit in person to purchase goods or services.

[0120] "Means for experiencing streetscapes of the past" refers to technologies and devices that allow users to visually experience streetscapes and buildings of the past through smart glasses.

[0121] The system for implementing the present invention includes smart glasses, a server, an analysis module, a generation module, and a display module. Detailed processing of each module and the hardware and software used will be described below.

[0122] 1. System Programming

[0123] The system includes a program that works in conjunction with smart glasses to analyze information from old maps and old photographs, generate a 3D model of the cityscape of the past, and display it to the user.

[0124] 2. Program Processing Explanation

[0125] Hardware used:

[0126] Smart glasses (e.g. Microsoft(R) HoloLens(R), Google(R) Glass(R))

[0127] Server (a computer with high-performance data processing capabilities)

[0128] Software used:

[0129] OpenCV: Image analysis library

[0130] ARModule: Augmented reality display module

[0131] OldMapAnalyzer: Old map analysis module

[0132] ThreeDModelGenerator: 3D model generation module

[0133] Process flow:

[0134] 1. Loading old maps:

[0135] The server uses OpenCV to read image data from old maps and old photographs, thereby obtaining past geographical information and architectural features as digital data.

[0136] 2. Feature Extraction:

[0137] The server uses the OldMapAnalyzer module to extract geographic and architectural features from old maps and photographs, providing detailed information about past streetscapes.

[0138] 3. 3D model generation:

[0139] The server uses the ThreeDModelGenerator module to generate a 3D model based on the extracted features, which allows the past streetscape and buildings to be reproduced in three dimensions.

[0140] 4.Display:

[0141] The server transmits the generated 3D model to the smart glasses via ARModule and displays it to the user, allowing the user to visually experience the streetscape of the past in a brick-and-mortar store.

[0142] 3. Specific Examples

[0143] For example, when a user visits a historical tourist site, they can wear the smart glasses and experience the townscape of the Edo period, allowing them to enjoy sightseeing while comparing modern scenery with past scenery.

[0144] Example prompt:

[0145] "Please recreate the streets of Asakusa in the Edo period. Analyze old maps and old photographs, generate a 3D model, and display it on the smart glasses."

[0146] In this way, users can enjoy shopping and sightseeing while comparing modern-day Asakusa with Asakusa of the Edo period.

[0147] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0148] Step 1:

[0149] The server uses OpenCV to read image data of old maps and old photographs. It receives the image file path of the old map or old photograph as input and obtains image data as output. This image data is used for subsequent analysis processing.

[0150] Step 2:

[0151] The server uses the OldMapAnalyzer module to extract geographic and architectural features from the loaded image data. As input, it receives the image data acquired in step 1, and as output, it generates a dataset containing geographic and architectural features. This dataset will be the basis for reconstructing the streetscape of the past.

[0152] Step 3:

[0153] The server uses the ThreeDModelGenerator module to generate a 3D model based on the extracted features. It receives as input the dataset containing the geographical and architectural features generated in step 2, and generates as output 3D model data. This 3D model data is used to reproduce the appearance of past streets and buildings in three dimensions.

[0154] Step 4:

[0155] The server sends the generated 3D model data to the smart glasses through ARModule. It receives the 3D model data generated in step 3 as input and sends data to be displayed on the smart glasses as output. This data is used for the user to visually experience the past cityscape through the smart glasses.

[0156] Step 5:

[0157] The user wears the smart glasses and experiences the streetscape of the past in a real store. As input, the user receives 3D model data displayed on the smart glasses, and as output, the user visually experiences the streetscape and buildings of the past. Through this experience, the user can enjoy shopping and sightseeing while comparing the modern landscape with the landscape of the past.

[0158] Example 2

[0159] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] In order to provide an experience that combines modern streetscapes with past streetscapes, technology is required to acquire appropriate historical data based on the user's current location, and analyze and generate it in real time. However, with conventional technology, there is no system that consistently acquires location information, analyzes historical data, and displays the generated streetscapes, making it difficult for users to smoothly experience past streetscapes.

[0161] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0162] In this invention, the server includes a means for acquiring an old map related to the vicinity of the location information, which is a map from a time before the present, and an old photograph related to the vicinity of the location information, based on the user's location information, a means for extracting geographical features and architectural features from the acquired old map and old photograph, a prompt sentence for instructing to generate a 3D model representing a landscape related to the vicinity of the location information, which is a landscape from a time before the present, based on the extracted geographical features and architectural features, a means for generating the 3D model based on a generative AI model, and a means for displaying an image obtained based on the generated 3D model on smart glasses worn by the user. This allows the user to smoothly enjoy an experience in which modern cityscapes and past cityscapes are blended in real time.

[0163] "Smart glasses" are wearable devices that, when worn by a user, can overlay digital information onto images of the real world.

[0164] “Location Information” means data that describes a User’s current geographic location, such as data obtained using GPS or other location measurement technologies.

[0165] A "server" is a computer system that processes and stores data via a network, and receives, analyzes, and generates information sent from a user's device.

[0166] An "antique map" is a map that shows geographical information from the past, and visually represents historical cityscapes and topography.

[0167] "Old photographs" are photographs taken in times past that record historical buildings and landscapes.

[0168] "Analysis" is the process of examining acquired data in detail and extracting useful information from it.

[0169] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate new data and content, generating past streetscapes based on a prompt text.

[0170] A "prompt" is an instruction entered into a generative AI model to cause it to generate a specific output.

[0171] "Streetscape" refers to the layout and appearance of buildings, roads, and other structures in a particular area.

[0172] "Display" refers to the process of visually presenting digital information to a user by overlaying the information on the smart glasses display.

[0173] This invention is a system that allows a user to wear smart glasses and stand at a specific location, and generates an old streetscape based on information from old maps and old photographs of the location, providing an experience that blends the old streetscape with the real world. A specific embodiment of this system will be described below.

[0174] First, a user puts on the smart glasses and stands at a specific location, for example, in Asakusa, Tokyo. The smart glasses use the built-in GPS sensor to obtain the user's current location information. This location information is then sent to the server in real time.

[0175] The server receives the location information sent from the device. Based on the received location information, it uses geographic information services such as Google Maps API and OpenStreetMap API to obtain old maps and old photographs of the location. The obtained data is analyzed within the server, and geographical and architectural features are abstracted.

[0176] Based on the analyzed data, the server generates a prompt sentence to be input to a generative AI model (e.g., GPT-4 (registered trademark) or DALL-E). For example, the server creates a prompt sentence such as, "Based on geographical features and architectural features, please generate a 3D model that represents the scenery of the Edo period around the current location."

[0177] The generative AI model generates a 3D model representing the old scenery (cityscape) based on the prompt sent from the server. The generated data is returned to the server. The server receives the 3D model data representing the old scenery returned from the generative AI model and sends it to the terminal.

[0178] The device receives 3D model data representing old scenery sent from the server. The images obtained from the received data are superimposed on the real-world image on the smart glasses display, allowing users to enjoy an experience in which the real world and old townscapes are blended in real time.

[0179] As a concrete example, consider the case where a user stands in front of Kaminarimon Gate in Asakusa. When the user wears smart glasses and stands at a specific point in Asakusa, the smart glasses send the location information of that point to the server. The server uses the Google Maps API to obtain old maps and old photos of Asakusa, and inputs the following prompt text to the generative AI model:

[0180]

[0181] The generative AI model generates an image of Asakusa in the Edo period based on this prompt and returns the data to the server. The server then sends the generated data of Asakusa in the Edo period to the device. The device then overlays the received data with real-world images and displays them on the smart glasses display. The user can experience an image that combines modern Asakusa with Asakusa in the Edo period.

[0182] This system allows users to smoothly enjoy an experience in real time where modern cityscapes blend with historic streetscapes.

[0183] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0184] Step 1:

[0185] A user puts on the smart glasses and stands at a particular point.

[0186] Input: User's location

[0187] Output: Smart glasses get location information

[0188] Specific operation: A user wears the smart glasses and stands at a specific location, for example, in Asakusa, Tokyo. The smart glasses use the built-in GPS sensor to obtain the user's current location information.

[0189] Step 2:

[0190] The device obtains location information and sends it to the server.

[0191] Input: Obtained location information

[0192] Output: Location sent to the server

[0193] How it works: The smart glasses transmit the acquired location information to the server in real time. The location information is sent to the server in the form of latitude and longitude.

[0194] Step 3:

[0195] The server receives the location information and retrieves the corresponding old maps and old photographs.

[0196] Input: Location sent

[0197] Output: Obtained old maps and old photographs

[0198] Specific operation: The server receives the location information sent from the device. Based on the received location information, it uses geographic information services such as Google Maps API and OpenStreetMap API to obtain old maps and old photos of that location.

[0199] Step 4:

[0200] The server analyzes the acquired data and inputs a prompt sentence along with the analysis results into the generative AI model.

[0201] Input: Obtained old maps and old photographs

[0202] Output: Prompt sentence to be input to the generative AI model

[0203] Specific operation: The server analyzes the acquired data of old maps and old photographs. Along with the analysis results, it generates a prompt sentence to be input to a generative AI model (e.g., GPT-4 or DALL-E). For example, it creates a prompt sentence such as, "Based on geographical features and architectural features, please generate a 3D model that represents the scenery of the Edo period around the current location."

[0204] Step 5:

[0205] The generative AI model generates a 3D model representing the old cityscape and returns the data to the server.

[0206] Input: prompt statement

[0207] Output: Generated 3D model data showing the old streetscape

[0208] Specific operation: The generative AI model generates old streetscapes based on the prompt text sent from the server. The generated data is returned to the server.

[0209] Step 6:

[0210] The server transmits the generated data to the terminal.

[0211] Input: Generated old streetscape data

[0212] Output: Data sent to the terminal

[0213] Specific operation: The server receives the data of old cityscapes returned by the generative AI model and sends it to the device.

[0214] Step 7:

[0215] The data received by the device is overlaid on images of the real world and displayed.

[0216] Input: Received data of old streetscapes

[0217] Output: Fusion image displayed on the smart glasses display

[0218] Specific operation: The device receives data of old streetscapes sent from the server, and displays the received data on the smart glasses display, overlaying it with images of the real world.

[0219] Step 8:

[0220] Users will experience a fusion of the real world and old-fashioned streetscapes.

[0221] Input: Fusion image displayed on the smart glasses display

[0222] Output: The user's experience

[0223] Specific operation: Through the smart glasses, the user experiences images that combine modern cityscapes with those of the past. For example, the user can enjoy images of modern Asakusa overlapping with those of Asakusa in the Edo period.

[0224] In addition, the user's emotion may be recognized using an emotion engine described later, and the display content of the smart glasses may be adjusted based on the recognized emotion of the user. Specifically, the color tone of the scenery and the facial expressions of people included in the displayed image may be adjusted according to the user's emotion.

[0225] (Application example 2)

[0226] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0227] Systems that allow users to visually experience a fusion of modern cityscapes and past cityscapes are extremely useful in the fields of tourism and education. However, with current technology, it is difficult for users to obtain, analyze, and visually display historical information about a specific location in real time when they stand there. In addition, technology that uses generative AI models to generate prompt sentences and display past cityscapes based on those sentences is still in its infancy. This poses the challenge of making it difficult for users to experience past cityscapes in real time.

[0228] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0229] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for acquiring information on old maps and old photographs of a specific location when the user stands at the location, a means for generating a prompt sentence using a generative AI model based on the acquired information, and a means for visually displaying the old streetscape based on the generated prompt sentence. This enables the user to experience the past streetscape of a specific location in real time simply by standing at the location.

[0230] "Smart glasses" are wearable devices that allow users to overlay digital information onto the real world.

[0231] An "antique map" is a map that shows geographical information from the past, recording historical topography and the layout of buildings.

[0232] "Old photographs" are photographs of landscapes, buildings, and people from the past, and serve as visual records of historical information.

[0233] "Means of analysis" refers to techniques and methods for processing the information from acquired old maps and old photographs as data and extracting geographical and architectural features.

[0234] "Means of generation" refers to the techniques and methods for recreating past streetscapes and landscapes as digital data based on analyzed information.

[0235] "Display means" refers to the technology or method for displaying the generated past cityscapes and scenery on the smart glasses.

[0236] A "particular location" is a location where a user is standing and about which historical information relating to that location is retrieved.

[0237] A "generative AI model" is a model that uses artificial intelligence to analyze data and generate output for a specific purpose.

[0238] A "prompt sentence" is an instruction sentence output by a generative AI model, and is text generated based on specific information.

[0239] "Means for visual display" refers to techniques or methods for displaying the generated past cityscapes or scenery so that the user can visually experience them.

[0240] The system for implementing the present invention includes smart glasses, a server, and a generative AI model. A specific embodiment of the system is described below.

[0241] First, when a user wears the smart glasses and stands at a specific location, the smart glasses send the location information of that location to the server. Based on the location information, the server retrieves information on old maps and old photographs related to that location from a database that stores past geographical information and historical photographs.

[0242] Next, the server analyzes the information on the acquired old maps and old photographs. Image processing techniques are used for the analysis to extract geographical and architectural features. Specifically, image processing libraries such as OpenCV are used to extract feature points from maps and photographs.

[0243] Based on the analyzed information, the server generates a prompt sentence using a generative AI model. For example, a natural language generation model such as GPT-3 (registered trademark) is used for the generative AI model. The generated prompt sentence is an instruction sentence for reproducing a past streetscape.

[0244] Based on the generated prompt, the server generates digital data of the streetscape of the past. This generation is done using 3D modeling software and rendering engines, such as Blender and Unity.

[0245] The generated past streetscapes are sent to the smart glasses and displayed in the user's field of view, allowing the user to have a visual experience that blends the real world with the past streetscapes.

[0246] As a concrete example, when a user stands at Tokyo Station, the server retrieves old photos of Tokyo Station, analyzes them, and generates a prompt. The generated prompt is "Please show me the old streetscape of Tokyo Station." Based on this prompt, the server generates an old streetscape of Tokyo Station and displays it on the smart glasses. The user can enjoy walking around while comparing the modern Tokyo Station with the old streetscape of Tokyo Station.

[0247] In this way, the present invention enables a user to experience a past streetscape of a particular location in real time, simply by standing at that location.

[0248] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0249] Step 1:

[0250] A user puts on the smart glasses and stands at a particular point.

[0251] Input: User's location

[0252] Output: Location information is sent from the smart glasses to the server.

[0253] Specific operation: The GPS function of the smart glasses obtains the user's current location and sends that location information to the server.

[0254] Step 2:

[0255] Based on the location information, the server retrieves information on old maps and old photographs related to that location from a database.

[0256] Input:Location

[0257] Output: Information on old maps and old photographs

[0258] Specific operation: The server queries the database using the location information as a key and retrieves the relevant data on old maps and old photographs.

[0259] Step 3:

[0260] The server analyzes the information on old maps and old photographs it has acquired.

[0261] Input: Information on old maps and old photographs

[0262] Output: Geographical and architectural features

[0263] Specific operation: The server uses image processing libraries such as OpenCV to extract feature points from old maps and old photographs.

[0264] Step 4:

[0265] The server generates a prompt sentence based on the parsed information using a generative AI model.

[0266] Input: Geographical and architectural features

[0267] Output: Prompt statement

[0268] Specific operation: The server inputs feature data into a generative AI model such as GPT-3 and generates prompt sentences to recreate past cityscapes.

[0269] Step 5:

[0270] The server generates past streetscapes as digital data based on the generated prompt sentence.

[0271] Input: Prompt statement

[0272] Output: Digital data of past streetscapes

[0273] Specific operation: The server uses 3D modeling software such as Blender or Unity to generate a 3D model based on the prompt text.

[0274] Step 6:

[0275] The server transmits the generated past streetscapes to the smart glasses and displays them in the user's field of view.

[0276] Input: Digital data of past streetscapes

[0277] Output: A past cityscape displayed on smart glasses

[0278] Specific operation: The server sends the generated 3D model to the smart glasses, which overlay it on the user's field of view.

[0279] In this way, users can have a visual experience that blends the real world with past cityscapes.

[0280] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0281] "Example 1"

[0282] The present invention is a system that works in conjunction with smart glasses, analyzes information from old maps and old photographs, generates old streetscapes based on the analyzed information, and displays the generated streetscapes on the smart glasses. It further includes an emotion engine that recognizes the user's emotions. This emotion engine estimates the user's emotions from the user's facial expressions, tone of voice, and operation patterns of the smart glasses. Based on the estimated emotions, the system adjusts the color tones and music of the generated streetscapes, the facial expressions of people displayed, and the like, to provide an experience that matches the user's emotions.

[0283] "Example 2"

[0284] As a specific example, when a user wears smart glasses and stands at a specific location, information on old maps and old photographs of that location is sent to the system. An analysis module in the system extracts geographical and architectural features from the sent information and generates old streetscapes based on the extracted features. The generated streetscape is displayed on the smart glasses, and the user experiences a fusion of the real world and old streetscapes. At this time, the emotion engine recognizes the user's emotions, such as joy, surprise, and nostalgia, and adjusts the display of the streetscape according to those emotions. For example, when the user is feeling joy, the colors of the streetscape are made vivid and the people's expressions are brightened. Conversely, when the user is feeling nostalgic, the colors of the streetscape are made subdued and the people's expressions are made gentler.

[0285] The process flow of each embodiment will be described below.

[0286] "Example 1"

[0287] Step 1: A user puts on the smart glasses and stands at a specific point.

[0288] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[0289] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[0290] Step 4: Generate old streetscapes based on the extracted features.

[0291] Step 5: Display the generated cityscape on the smart glasses.

[0292] Step 6: The emotion engine recognizes the user's emotion and adjusts the display of the cityscape according to that emotion.

[0293] "Example 2"

[0294] Step 1: A user puts on the smart glasses and stands at a specific point.

[0295] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[0296] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[0297] Step 4: Generate old streetscapes based on the extracted features.

[0298] Step 5: Display the generated cityscape on the smart glasses.

[0299] Step 6: The emotion engine recognizes the user's emotion and adjusts the display of the streetscape according to that emotion. For example, when the user is feeling happy, the colors of the streetscape are made vivid and the people's expressions brighter. On the other hand, when the user is feeling nostalgic, the colors of the streetscape are made muted and the people's expressions calmer.

[0300] Example 1

[0301] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0302] With conventional technology, it was difficult to analyze information from old maps and old photographs and recreate old streetscapes. In addition, it was not possible to adjust the display content according to the user's emotions, making it impossible to provide a consistent experience to the user. Furthermore, there was a lack of technology to change the display content in real time in conjunction with smart glasses. This led to problems such as a decrease in the sense of immersion and satisfaction when users experienced historical streetscapes.

[0303] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0304] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for recognizing a user's emotion, and a means for adjusting the display content based on the recognized emotion. This allows a user to experience old streetscapes in real time based on information on old maps and old photographs, and further allows the experience to be adjusted according to the user's emotion.

[0305] "Smart glasses" are wearable devices that can display visual information as augmented reality when worn by a user.

[0306] An "antique map" is a map that shows geographical information from the past, including historical geographical features and the layout of buildings.

[0307] "Old photographs" are photographs taken in the past that record historical buildings and landscapes.

[0308] An "analysis module" is software or hardware used to analyze information from old maps and old photographs and extract geographical and architectural features.

[0309] A "3D model" is a three-dimensional digital model that is generated based on analyzed geographical features and architectural characteristics.

[0310] An "emotion engine" is software or hardware that recognizes and estimates a user's emotions from the user's facial expressions, tone of voice, operation patterns, etc.

[0311] The "means for adjusting the display content" refers to software or hardware for changing the color tones of the displayed streetscape, the music, the facial expressions of people displayed, etc., based on the recognized emotions of the user.

[0312] "Geographic features" refers to geographic elements such as rivers, roads, and mountains on a map or photograph.

[0313] "Building features" refers to elements such as the shape, layout, and structure of a building on a map or photograph.

[0314] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0315] This invention is a system that works in conjunction with smart glasses, analyzes information from old maps and old photographs, generates old streetscapes based on the analyzed information, and displays the generated streetscapes on the smart glasses. It also includes a function to recognize the user's emotions and adjust the display content based on the emotions.

[0316] Hardware and Software Used

[0317] The server uses the following hardware and software:

[0318] Image analysis library: OpenCV, TensorFlow

[0319] 3D modeling software: Unity, Unreal Engine

[0320] Emotion recognition API: Microsoft Azure (registered trademark) Emotion API, Google Cloud Speech-to-Text API

[0321] Communication technology: Wi-Fi, Bluetooth

[0322] Data processing and calculation

[0323] The server receives data of old maps and old photographs uploaded by users. The received data is analyzed using image analysis libraries (OpenCV and TensorFlow). The analysis module extracts geographical features (rivers, roads, mountains, etc.) and architectural features (shape, layout, structure, etc. of buildings).

[0324] Based on the analysis results, the server uses 3D modeling software (Unity or Unreal Engine) to generate a 3D model. The generated 3D model reproduces the old streetscape. For example, when recreating a streetscape from the Edo period, the layout of the buildings and roads at that time is faithfully reproduced.

[0325] The generated 3D model is then sent to the smart glasses via Wi-Fi or Bluetooth, where the smart glasses display the received 3D model, giving the user the experience of traveling back in time.

[0326] Emotion recognition and display content adjustment

[0327] The server recognizes the user's emotions using an emotion engine. The emotion engine estimates the user's emotions from the user's facial expressions, tone of voice, and operation patterns of the smart glasses. For example, it uses Microsoft Azure's Emotion API or Google Cloud's Speech-to-Text API.

[0328] After the emotion engine estimates the user's emotion, the server adjusts the display content based on that emotion. For example, if the server estimates that the user is having fun, it will brighten the colors of the streetscape and play cheerful music. It will also adjust the facial expressions of people displayed to smile.

[0329] Examples of concrete examples and prompts

[0330] For example, if a user uploads old maps and old photographs from the Edo period, the server receives this data and uses an analysis module to extract geographical and architectural features from the Edo period. Based on the analysis results, a 3D model of the Edo period streetscape is generated using Unity. The generated 3D model is sent to the smart glasses via Wi-Fi, and the user can experience the Edo period streetscape through the smart glasses.

[0331] When a user is having fun walking through a street, the emotion engine infers from the user's facial expression that the user is having fun. The server brightens the colors of the street and plays cheerful music. The server also adjusts the facial expressions of the people displayed to smile.

[0332] Example of a prompt to be input to a generative AI model:

[0333] "Analyze old maps and photographs from the Edo period and generate a 3D model of the streets from that period. When the user is enjoying it, brighten the colors of the streets and play fun music."

[0334] In this way, a system is realized in which the server, terminal, and user work together to provide the user with an experience of historical townscapes that correspond to their emotions.

[0335] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0336] Step 1:

[0337] Users upload old maps and old photographs.

[0338] A user uploads data of old maps and old photographs using a terminal (e.g., a smartphone or tablet) connected to the smart glasses. As input, they provide image files in JPEG or PNG format. As output, the uploaded data is sent to the server.

[0339] Step 2:

[0340] The server receives the uploaded data.

[0341] The server receives data of old maps and old photographs uploaded by users. As input, it receives image files sent by users. As output, it stores the received data in a database in the server.

[0342] Step 3:

[0343] The server analyzes the data using an analysis module.

[0344] The server starts the analysis module and analyzes the received data of old maps and old photographs. It uses the stored image files as input. The analysis module uses image analysis libraries such as OpenCV and TensorFlow to extract geographical and architectural features. The extracted feature information is obtained as output.

[0345] Step 4:

[0346] The server generates a 3D model based on the analysis results.

[0347] The server uses 3D modeling software such as Unity or Unreal Engine to generate a 3D model based on the feature information obtained from the analysis module. It uses the extracted feature information as input. As output, a 3D model that reproduces the old streetscape is generated.

[0348] Step 5:

[0349] The server transmits the generated 3D model to the smart glasses.

[0350] The server sends the generated 3D model to the smart glasses. As input, the server uses the generated 3D model. As output, the 3D model is sent to the smart glasses. For transmission, wireless communication technologies such as Wi-Fi and Bluetooth are used.

[0351] Step 6:

[0352] Smart glasses display 3D models.

[0353] The smart glasses display the 3D model received from the server. As input, they use the 3D model sent from the server. As output, the user can visually experience the old cityscape as if they had traveled back in time.

[0354] Step 7:

[0355] Users experience the cityscape through smart glasses.

[0356] A user can wear the smart glasses and walk around the displayed 3D model of the cityscape. As input, we use the 3D model displayed on the smart glasses. As output, the user experiences the historical cityscape.

[0357] Step 8:

[0358] The server uses an emotion engine to recognize the user's emotion.

[0359] The server starts an emotion engine and recognizes the user's emotions from the user's facial expressions, tone of voice, and operation patterns of the smart glasses. The inputs used are the user's facial expression data, voice data, and operation data. The emotion engine uses Microsoft Azure's Emotion API and Google Cloud's Speech-to-Text API. The output is an estimated user's emotions.

[0360] Step 9:

[0361] The server adjusts the display content based on the user's emotions.

[0362] The server adjusts the display content of the generated 3D model based on the user's emotions estimated by the emotion engine. As input, it uses the estimated user's emotion data. As output, the color tones of the streetscape, music, and facial expressions of people displayed are adjusted. For example, if it is estimated that the user is having fun, the color tones of the streetscape are brightened and cheerful music is played. The facial expressions of people displayed are also adjusted to smile.

[0363] (Application example 1)

[0364] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0365] Conventional smart glasses-based systems can analyze old maps and old photographs to recreate old streetscapes, but they cannot adjust the display content according to the user's emotions, so the user experience is uniform and it is difficult to provide an optimized experience for each individual user. In addition, there is a lack of concrete measures to improve the shopping experience in physical stores. This makes it difficult to increase user satisfaction and stimulate purchasing desire in physical stores.

[0366] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0367] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for recognizing the user's emotions, and a means for adjusting the display content based on the recognized emotions. This makes it possible to provide optimal display content according to the user's emotions, improving the shopping experience in a brick-and-mortar store and increasing the user's satisfaction and purchasing motivation.

[0368] "Smart glasses" are wearable devices that can display visual information as augmented reality when worn by the user.

[0369] An "antique map" is a map that shows geographical information from the past, recording the locations of historical geographical features and buildings.

[0370] "Old photographs" are photographs taken in the past that record historical buildings and landscapes.

[0371] "Means of analysis" refer to the technical means for analyzing information from old maps and old photographs as digital data and extracting geographical and architectural features.

[0372] The "means of generation" refers to the technical means for generating a three-dimensional model of a past streetscape based on the analyzed information.

[0373] "Means for displaying" means the technical means for displaying the generated 3D model on the smart glasses and providing visual information to the user.

[0374] "Means of recognizing emotions" refers to technical means for inferring emotions from a user's facial expressions, tone of voice, operation patterns, etc.

[0375] "Means for adjusting the display content" refers to technical means for adjusting the color tones of the displayed cityscape, the music, the facial expressions of people displayed, etc., based on the recognized emotions of the user.

[0376] The system for implementing this invention uses hardware such as smart glasses, a server, a camera, a microphone, etc., and software such as OpenCV, TensorFlow, Affectiva SDK, Unity, FMOD, etc. The specific configuration and operation of the system are described below.

[0377] System Configuration

[0378] 1. Smart glasses:

[0379] It is a wearable device that is worn by the user and displays visual information as augmented reality.

[0380] 2. Server:

[0381] It is a central processing device that analyzes information from old maps and old photographs and generates old cityscapes based on the analyzed information.

[0382] 3. Camera:

[0383] A device for capturing a user's facial expressions.

[0384] 4. Microphone:

[0385] A device for capturing the tone of a user's voice.

[0386] Software configuration

[0387] 1. OpenCV:

[0388] This is a library for analyzing images of old maps and old photographs.

[0389] 2. TensorFlow:

[0390] A framework for running machine learning models for image analysis.

[0391] 3. Affectiva SDK:

[0392] It is a software development kit for recognizing user emotions.

[0393] 4. Unity:

[0394] It is a game engine for generating 3D models and adjusting the display content.

[0395] 5. FMOD:

[0396] An audio engine for controlling music and sound effects.

[0397] System Operation

[0398] 1. Analysis of old maps and photographs:

[0399] The server uses OpenCV and TensorFlow to analyze old maps and photographs and extract geographical and architectural features.

[0400] 2. Generate 3D model:

[0401] The server uses Unity to generate a 3D model of the old townscape based on the extracted features.

[0402] 3. Emotion Recognition:

[0403] The server analyzes the data obtained from the camera and microphone using the Affectiva SDK to recognize the user's emotions.

[0404] 4. Adjust the display:

[0405] The server uses Unity and FMOD to adjust the content displayed based on the emotions it recognizes, such as changing the color tones of the cityscape, the music, and the facial expressions of people displayed.

[0406] Examples

[0407] If a user wants to experience the streets of the Edo period, they can wear the smart glasses and walk around a brick-and-mortar store. The streets of the Edo period, recreated from old maps and photographs, are displayed on the smart glasses, giving the user the experience of traveling back in time to that era. Furthermore, if the user has a happy expression, the colors of the streets become brighter and cheerful music plays.

[0408] Examples of prompt statements

[0409] "Develop an application for smart glasses that recreates the streets of the Edo period and adjusts the display content according to the user's emotions. Analyze old maps and old photographs, generate 3D models, and use an emotion recognition engine to adjust the display content."

[0410] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0411] Step 1:

[0412] The server receives image data of old maps and old photographs. As input, image files of old maps and old photographs are provided. The server reads these images using OpenCV and performs preprocessing. Specifically, it removes noise from the images and adjusts the resolution. As output, it obtains the preprocessed image data.

[0413] Step 2:

[0414] The server inputs the preprocessed image data into a TensorFlow model to extract geographic and architectural features. The preprocessed image data is provided as input. The server uses TensorFlow to perform image analysis and extract features. Specifically, it identifies building outlines and topographical features. The extracted feature data is obtained as output.

[0415] Step 3:

[0416] The server uses Unity to generate a 3D model based on the extracted feature data. The feature data is provided as input. The server executes Unity scripts to generate a 3D model that reproduces the past streetscape. Specifically, it reproduces the layout of buildings and the terrain. The generated 3D model data is obtained as output.

[0417] Step 4:

[0418] The user wears the smart glasses and transmits real-time data to the server through the camera and microphone. As input, the user's facial expression data and voice data are provided. The server uses the Affectiva SDK to analyze these data and recognize the user's emotions. Specifically, it analyzes changes in facial expressions and tone of voice. As output, it obtains the recognized emotion data.

[0419] Step 5:

[0420] The server uses Unity and FMOD to adjust the display content based on the recognized emotion data. Emotion data and 3D model data are provided as input. The server executes Unity scripts to adjust the color tone of the cityscape, the music, and the facial expressions of the people displayed. Specifically, it brightens the color tone and changes the music depending on the emotion. The adjusted display content data is obtained as output.

[0421] Step 6:

[0422] The server transmits the adjusted display content data to the smart glasses to display it to the user. As an input, the adjusted display content data is provided. The server transmits data to the display of the smart glasses to provide visual information to the user. Specifically, the reproduced cityscape and the adjusted display content are displayed on the smart glasses. As an output, visual information experienced by the user is obtained.

[0423] Example 2

[0424] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0425] In conventional systems, it was difficult for a user to visually reproduce the past streetscape of a particular location when the user stood at that location. In addition, the display content could not be dynamically adjusted according to the user's emotions, limiting the user experience. This led to the issue that users could not experience the past streetscape more realistically.

[0426] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0427] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, and a means for recognizing the user's emotions and adjusting the display according to the emotions. This makes it possible to visually reproduce the past streetscape of a specific location when the user stands at that location in real time, and further dynamically adjust the display content according to the user's emotions.

[0428] "Smart glasses" are wearable devices that, when worn by a user, can overlay digital information onto visual information from the real world.

[0429] An "antique map" is a map that shows geographical information from the past, and includes historical geographical information for specific locations or areas.

[0430] An "archive photograph" is a photograph taken in the past that contains historical visual information of a particular location or area.

[0431] "Means of analysis" refers to the technical means for receiving information from old maps and old photographs and extracting geographical and architectural features from that information.

[0432] The "means of generation" refers to the technical means for recreating the streetscape of the past based on the analyzed information, specifically the generation of a three-dimensional model.

[0433] The "display means" refers to the technical means for displaying the generated past streetscape on the smart glasses.

[0434] "Means for recognizing emotions" refers to a technical means for detecting emotions from a user's facial expressions, voice, etc., and adjusting the display content based on those emotions.

[0435] This invention is a system in which a user wears smart glasses and stands at a specific location, which visually recreates the past streetscape of that location, and dynamically adjusts the display content according to the user's emotions.

[0436] The server receives information on old maps and old photos sent from the smart glasses when a user stands at a specific location. The received information is passed to the analysis module, which extracts geographical and architectural features. The analysis module generates old streetscapes based on the extracted features. A generative AI model is used for this generation. The generated streetscape data is then sent to the smart glasses.

[0437] When a user stands at a specific spot, the device (smart glasses) sends information about old maps and old photographs of that spot to the server. The device receives data on old streetscapes generated by the server and displays it blended with the real world. The device also uses an emotion engine to recognize the user's emotions and adjusts the display according to those emotions. For example, if the user is feeling happy, the colors of the streetscape will be made more vivid and the people's expressions will be brighter. Conversely, if the user is feeling nostalgic, the colors of the streetscape will be made more subdued and the people's expressions will be made calmer.

[0438] The user puts on the smart glasses and stands at a specific location. The glasses send information about old maps and old photos of that location to the server, and receive the old streetscape generated by the server. The user can enjoy an experience where the real world and the old streetscape are fused together. In addition, the user's emotions are recognized and the display is adjusted according to those emotions, providing a more moving experience.

[0439] As a concrete example, let's say a user is standing in Asakusa, Tokyo. When the user wears smart glasses and stands in front of Kaminarimon Gate in Asakusa, the smart glasses send information about old maps and old photographs of that location to the server. The server analyzes the received information and generates the streetscape of Asakusa in the Edo period. The generated streetscape data is sent to the smart glasses as a 3D model. The smart glasses display the Edo period streetscape superimposed on the real scenery in front of Kaminarimon Gate. If the user is feeling happy, the colors of the Edo period streetscape will become more vivid and people's expressions will become brighter. On the other hand, if the user is feeling nostalgic, the colors of the streetscape will become more subdued and people's expressions will become calmer.

[0440] Examples of prompts to be input to a generative AI model include the following:

[0441] "When a user stands in Asakusa, Tokyo, generate an image of the streets of Asakusa from the Edo period and display it on the smart glasses. Adjust the display according to the user's emotions. If the user is feeling joy, make the colors vivid and brighten the people's expressions. If the user is feeling nostalgia, make the colors calmer and calm the people's expressions."

[0442] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0443] Step 1:

[0444] A user puts on the smart glasses and stands at a specific point.

[0445] Specific operation: The user wears the smart glasses and stands in front of the Kaminarimon Gate in Asakusa, Tokyo, for example. The smart glasses acquire the user's location information.

[0446] Input: User's location

[0447] Output: Get location information

[0448] Step 2:

[0449] The terminal (smart glasses) sends information about old maps and old photographs of that location to the server.

[0450] How it works: The smart glasses use GPS and Wi-Fi location information to determine the user's current location and send that information to a server.

[0451] Input: User's location

[0452] Output: Send location information

[0453] Step 3:

[0454] The server passes the received information to an analysis module, which extracts geographical and architectural features.

[0455] Specific operation: The server receives information on old maps and old photographs sent from the terminal and passes it to the analysis module. The analysis module extracts geographical features (e.g., road layout and river location) and architectural features (e.g., building shape and height).

[0456] Input: Information on old maps and old photographs

[0457] Output: Extraction of geographic and architectural features

[0458] Step 4:

[0459] The server generates old streetscapes based on the extracted features.

[0460] Specific operation: Based on the features extracted by the analysis module, the server uses a generative AI model to generate old streetscapes. For example, it recreates the streetscape of Asakusa in the Edo period.

[0461] Input: Geographical and architectural features

[0462] Output: Generation of old streetscape

[0463] Step 5:

[0464] The server transmits the generated streetscape data to the terminal.

[0465] Specific operation: The server sends the generated data of the old cityscape to the terminal as a 3D model or image data.

[0466] Input: Data of old streetscape

[0467] Output: Sending streetscape data

[0468] Step 6:

[0469] The streetscape data received by the device is blended with the real world and displayed.

[0470] Specific operation: The device overlays the received streetscape data onto the real-world scenery, and displays the fusion of real-world scenery and the old streetscape on the smart glasses display.

[0471] Input: Streetscape data

[0472] Output: Real-world blended display

[0473] Step 7:

[0474] The device recognizes the user's emotions using an emotion engine and adjusts the display accordingly.

[0475] Specific operation: The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, if the user is feeling happy, the colors of the streetscape will be brighter and the people's expressions will be brighter. On the other hand, if the user is feeling nostalgic, the colors of the streetscape will be muted and the people's expressions will be calmer.

[0476] Input: User emotion data

[0477] Output: Adjusting display according to emotions

[0478] (Application example 2)

[0479] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0480] In modern brick-and-mortar stores, there is a lack of ways to allow customers to visually experience past history and culture. In addition, technology to dynamically adjust display content according to customer emotions is also underdeveloped. This makes it difficult to increase customer interest and satisfaction.

[0481] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0482] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for recognizing a user's emotion, and a means for adjusting the display content according to the recognized emotion. This allows customers to visually experience past history and culture in a brick-and-mortar store, and the experience can be dynamically adjusted according to the customer's emotion.

[0483] "Smart glasses" are glasses-type devices that users can wear to experience augmented reality (AR) and virtual reality (VR).

[0484] An "antique map" is a map that shows geographical information from the past, and records historical cityscapes and topography.

[0485] "Old photographs" are photographs taken in the past that record historical buildings and landscapes.

[0486] "Means of analysis" refers to the technology and equipment required to process information from old maps and old photographs as digital data and to understand and analyze their contents.

[0487] "Means of generation" refers to technology and devices for digitally recreating past cityscapes and landscapes based on analyzed information.

[0488] "Display means" refers to the technology and devices for projecting the generated past cityscapes and scenery onto the display of smart glasses.

[0489] "Means of recognizing emotions" refers to technology or devices for detecting and analyzing emotions from a user's facial expressions, voice, etc.

[0490] "Means for adjusting the display content" refers to technology or devices for dynamically changing the color tone, brightness, etc. of the displayed past cityscapes or scenery in response to the recognized emotions of the user.

[0491] A system for implementing the present invention includes smart glasses, a server, and various software modules. Specific embodiments are described below.

[0492] System Configuration

[0493] The system includes smart glasses, a server, and the following software modules:

[0494] Image processing library (e.g. OpenCV)

[0495] Emotion recognition library (e.g. EmotionRecognizer)

[0496] Augmented reality display library (e.g. ARDisplay)

[0497] Custom modules for obtaining old maps and old photographs (e.g. historical_data)

[0498] Hardware

[0499] Smart glasses (e.g. Microsoft HoloLens)

[0500] Smart glasses with built-in camera

[0501] software

[0502] OpenCV: Image processing library

[0503] EmotionRecognizer: Emotion recognition library

[0504] ARDisplay: an augmented reality display library

[0505] historical_data: A custom module for retrieving historical maps and photographs.

[0506] Processing flow

[0507] 1. Acquire video from the smart glasses camera

[0508] The camera built into the smart glasses captures footage in real time.

[0509] 2. Get current location information

[0510] The GPS function of the smart glasses is used to obtain the user's current location.

[0511] 3. Obtain information on old maps and old photographs

[0512] Based on the acquired location information, the server retrieves corresponding data on old maps and old photographs from the historical_data module.

[0513] 4. Emotion recognition

[0514] The server uses the video captured by the smart glasses' camera to recognize the user's emotions using the EmotionRecognizer library.

[0515] 5. Adjusting the display content

[0516] Depending on the recognized emotion, the ARDisplay library is used to adjust the color tone and brightness of the displayed content (old maps and old photographs).

[0517] 6. AR display

[0518] Adjusted old maps and old photographs are overlaid on the display of the smart glasses.

[0519] Examples

[0520] When a user enters a traditional Japanese sweets shop, the smart glasses will show the shop's original appearance and manufacturing process from the past. If the user is feeling happy, the images displayed will be bright and vivid. If the user is feeling nostalgic, the images will be muted.

[0521] Examples of prompt statements

[0522] "When a user puts on the smart glasses and enters a long-established Japanese confectionery shop, please develop an application that displays the shop's history and the manufacturing process from the past. Please also add a function to adjust the display content according to the user's emotions."

[0523] The above is a specific embodiment for carrying out the present invention.

[0524] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0525] Step 1:

[0526] The server acquires video in real time from the smart glasses' camera.

[0527] Input: Video data from the camera built into the smart glasses

[0528] Output: Real-time video frames

[0529] Specific operation: The camera in the smart glasses captures the user's field of vision and sends the video data to the server.

[0530] Step 2:

[0531] The server obtains the user's current location information using the smart glasses' GPS function.

[0532] Input: GPS data from smart glasses

[0533] Output: Current location information (latitude and longitude)

[0534] Specific operation: The GPS sensor in the smart glasses obtains the current location information and sends the data to the server.

[0535] Step 3:

[0536] Based on the acquired location information, the server retrieves corresponding data on old maps and old photographs from the historical_data module.

[0537] Input: Current location

[0538] Output: Data of old maps and old photographs

[0539] Specific operation: The server sends a query to the historical_data module based on the location information and retrieves the relevant data on old maps and old photographs.

[0540] Step 4:

[0541] The server uses the video captured by the smart glasses' camera to recognize the user's emotions using the EmotionRecognizer library.

[0542] Input: Real-time video frames

[0543] Output: User emotion data (e.g., happiness, nostalgia)

[0544] Specific operation: The server inputs video frames into the EmotionRecognizer library and analyzes emotions from the user's facial expressions and voice.

[0545] Step 5:

[0546] The server uses the ARDisplay library to adjust the color and brightness of the displayed content (old maps and old photographs) according to the recognized emotion.

[0547] Input: User emotion data, data on old maps and old photographs

[0548] Output: Adjusted data of historical maps and photographs

[0549] Specific operation: Based on the emotion data, the server dynamically changes the color tone and brightness of old maps and old photographs using the ARDisplay library.

[0550] Step 6:

[0551] The server then overlays the adjusted old maps and old photographs onto the smart glasses' display.

[0552] Input: Adjusted historical map and photo data, real-time video frames

[0553] Output: Augmented reality video displayed on smart glasses

[0554] Specific operation: The server overlays the adjusted old maps and old photos onto real-time video and sends them to the smart glasses display for display.

[0555] The above are the specific process steps for carrying out the present invention.

[0556] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0557] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0558] Another example of generative AI is Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0559] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0560] [Second embodiment]

[0561] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0562] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0563] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0564] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0565] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0566] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0567] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0568] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0569] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0570] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0571] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0572] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0573] "Example 1"

[0574] As one embodiment of the present invention, a system that works with smart glasses is provided. The system includes a module that analyzes information on old maps and old photographs. The analysis module extracts geographical features and architectural features from the old maps and old photographs. A module that generates old streetscapes operates based on the extracted features. The generation module generates a 3D model based on the extracted geographical features and architectural features. The generated streetscape is displayed on the smart glasses, allowing the user to experience a time-traveling experience.

[0575] "Example 2"

[0576] For example, when a user wears smart glasses and stands at a specific location, information about old maps and old photographs of that location is sent to the system. The system analyzes the information and generates an old streetscape of that location. The generated streetscape is displayed on the smart glasses, allowing the user to experience a fusion of the real world and the old streetscape.

[0577] The process flow of each embodiment will be described below.

[0578] "Example 1"

[0579] Step 1: A user puts on the smart glasses and stands at a specific point.

[0580] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[0581] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[0582] Step 4: The module that generates old streetscapes runs based on the extracted features.

[0583] Step 5: The generated streetscape is displayed on the smart glasses, giving the user an experience that blends the real world with the old streetscape.

[0584] "Example 2"

[0585] Step 1: A user puts on the smart glasses and stands at a specific point.

[0586] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[0587] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[0588] Step 4: The module that generates old streetscapes runs based on the extracted features.

[0589] Step 5: The generated streetscape is displayed on the smart glasses, giving the user an experience that blends the real world with the old streetscape.

[0590] Example 1

[0591] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0592] With conventional technology, it was difficult to reproduce streetscapes of the past using old maps and old photographs. In addition, the means by which users could experience streetscapes of the past were limited. This restricted the use of the technology in fields such as the reproduction of historical streetscapes, education, and tourism.

[0593] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0594] In this invention, the server includes a means for a user to upload old maps and old photographs from a terminal, a means for the server to receive the uploaded data, a means for the server to analyze the data using an analysis module and extract geographical features and architectural features, a means for the server to generate a 3D model using a generation module based on the features extracted by the server, a means for the server to transmit the generated 3D model to the smart glasses, and a means for the smart glasses to display the 3D model and for the user to experience it. This allows the user to experience past streetscapes in real time.

[0595] A "user" is an individual or group that uses the system to upload old maps and old photographs and experience streetscapes of the past.

[0596] A "terminal" is a device that a user uses to upload old maps and old photographs, and includes smartphones, personal computers, etc.

[0597] "Server" refers to a computer system that receives data uploaded by users and performs analysis and generation of 3D models.

[0598] The "analysis module" is a software component that analyzes data from old maps and old photographs and extracts geographical and architectural features.

[0599] "Geographical features" are geographical elements such as rivers, roads, and mountains that are included in old maps and old photographs.

[0600] "Building characteristics" refers to information such as the shape and layout of buildings contained in old maps and photographs.

[0601] The "generative module" is a software component for generating 3D models based on extracted geographical and architectural features.

[0602] A "3D model" is a three-dimensional digital representation generated based on extracted features.

[0603] "Smart glasses" are a wearable device that allows users to visually experience three-dimensional models.

[0604] "Uploading" refers to the act of a user using a terminal to send data of old maps or old photographs to a server.

[0605] "Receiving" is the act of the server taking in data sent by the user.

[0606] "Transmitting" refers to the act of the server sending the generated 3D model to the smart glasses.

[0607] "Display" refers to the act of the smart glasses projecting the received 3D model into the user's field of vision.

[0608] An "experience" is an act in which a user can visually sense a streetscape of the past through the smart glasses.

[0609] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0610] This invention is a system that allows users to experience past streetscapes using old maps and old photographs. This system includes a series of processes in which the user uploads old maps and old photographs from a terminal, the server analyzes the data, generates a 3D model, and displays it to the user through smart glasses.

[0611] Hardware and Software Used

[0612] Terminal

[0613] Users upload old maps and old photographs using devices such as smartphones and computers, allowing them to easily access the system and provide data.

[0614] server

[0615] The server receives data uploaded by users, analyzes it, and generates 3D models. The following software is installed on the server:

[0616] Analysis module: Uses Python's OpenCV library and machine learning frameworks such as TensorFlow to analyze data from old maps and photographs and extract geographic and architectural features.

[0617] Generation module: Using 3D modeling software such as Blender or Unity, a 3D model is generated based on the extracted features.

[0618] Smart Glasses

[0619] The smart glasses display the 3D model sent from the server in the user's field of vision, allowing the user to experience past streetscapes in real time.

[0620] Examples

[0621] For example, if a user uploads an old map of Tokyo in the Edo period in JPEG format, the server receives this data and uses OpenCV to detect the outlines of rivers, roads, and buildings.The server then uses Blender to generate a 3D model of the streets of Tokyo in the Edo period based on the detected features.This 3D model is sent to the smart glasses via Wi-Fi, allowing the user to experience Tokyo in the Edo period through the smart glasses.

[0622] Examples of prompt statements

[0623] Examples of prompts to be input to a generative AI model include the following:

[0624] "Analyze old maps of Tokyo from the Edo period and generate a 3D model of the cityscape from that time."

[0625] Using this prompt, the generative AI model can perform the necessary analysis and generation processes to provide the user with an experience that recreates a historical cityscape.

[0626] In this way, the present invention provides a system that allows a user to experience past streetscapes in real time.

[0627] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0628] Program processing flow

[0629] Step 1: User uploads old maps and old photos from their device

[0630] Users upload data on old maps and old photographs to the system using devices such as smartphones and computers. For example, an old map of Tokyo from the Edo period can be uploaded in JPEG format.

[0631] Input: Data of old maps and old photographs (e.g. JPEG format images)

[0632] Output: Uploaded data is sent to the server

[0633] Step 2: The server receives the uploaded data

[0634] The server receives data of old maps and old photographs uploaded by users. This data is stored in a specific directory on the server.

[0635] Input: Data on old maps and old photographs sent by users

[0636] Output: Data stored in a specific directory on the server

[0637] Step 3: The server analyzes the data using the analysis module to extract geographic and architectural features.

[0638] The server uses machine learning frameworks such as Python's OpenCV library and TensorFlow to analyze the uploaded data, for example to detect the outlines of rivers, roads, and buildings in old maps and extract their features.

[0639] Input: Data of old maps and old photographs stored on the server

[0640] Output: Extracted geographic and architectural features (e.g. rivers, roads, building outlines)

[0641] Step 4: Based on the features extracted by the server, a 3D model is generated using the generation module.

[0642] The server uses 3D modeling software such as Blender or Unity to generate 3D models based on the extracted features, for example recreating the streets of Edo-period Tokyo based on the location information of detected rivers and roads.

[0643] Input: Extracted geographic and building features

[0644] Output: Generated 3D model

[0645] Step 5: The server sends the generated 3D model to the smart glasses

[0646] The server then transmits the generated 3D model to the smart glasses via wireless communication such as Wi-Fi or Bluetooth.

[0647] Input: Generated 3D model

[0648] Output: 3D model sent to smart glasses

[0649] Step 6: The smart glasses display the 3D model and the user experiences it.

[0650] The smart glasses then display the received 3D model in the user's field of vision, giving the user the experience of being transported back in time to Edo-period Tokyo.

[0651] Input: 3D model sent to smart glasses

[0652] Output: A 3D model displayed in the user's field of view.

[0653] In this way, the system allows users to experience past streetscapes in real time.

[0654] (Application example 1)

[0655] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0656] In modern brick-and-mortar stores, it is difficult for users to experience the streetscapes and buildings of the past. This limits opportunities to understand historical background and culture. In addition, there is a lack of ways to visually reproduce past scenery in tourist spots and historical places, making it difficult to attract users' interest.

[0657] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscape on the smart glasses, and a means for allowing the user to experience past streetscapes in a real store. This allows the user to visually experience the appearance of past streetscapes and buildings in a real store, and to gain a deeper understanding of the historical background and culture.

[0658] "Smart glasses" are glasses-type devices that users can wear to provide augmented reality and display information.

[0659] An "antique map" is a map that shows geographical information from the past, recording historical geographical features and the layout of buildings.

[0660] "Old photographs" are photographs of landscapes, buildings, and people from the past, providing visual historical information.

[0661] "Means of analysis" refers to technologies and devices for digitally processing information from old maps and old photographs and extracting geographical and architectural features.

[0662] "Means of generation" refers to technology and equipment for creating 3D models of past cityscapes and buildings based on analyzed information.

[0663] "Display means" refers to the technology or devices for displaying the generated 3D model on the smart glasses and providing the user with a visual experience.

[0664] A "brick and mortar store" is a store that exists in a physical location and where users can visit in person to purchase goods or services.

[0665] "Means for experiencing streetscapes of the past" refers to technologies and devices that allow users to visually experience streetscapes and buildings of the past through smart glasses.

[0666] The system for implementing the present invention includes smart glasses, a server, an analysis module, a generation module, and a display module. Detailed processing of each module and the hardware and software used will be described below.

[0667] 1. System Programming

[0668] The system includes a program that works in conjunction with smart glasses to analyze information from old maps and old photographs, generate a 3D model of the cityscape of the past, and display it to the user.

[0669] 2. Program Processing Explanation

[0670] Hardware used:

[0671] Smart glasses (e.g. Microsoft HoloLens, Google Glass)

[0672] Server (a computer with high-performance data processing capabilities)

[0673] Software used:

[0674] OpenCV: Image analysis library

[0675] ARModule: Augmented reality display module

[0676] OldMapAnalyzer: Old map analysis module

[0677] ThreeDModelGenerator: 3D model generation module

[0678] Process flow:

[0679] 1. Loading old maps:

[0680] The server uses OpenCV to read image data from old maps and old photographs, thereby obtaining past geographical information and architectural features as digital data.

[0681] 2. Feature Extraction:

[0682] The server uses the OldMapAnalyzer module to extract geographic and architectural features from old maps and photographs, providing detailed information about past streetscapes.

[0683] 3. 3D model generation:

[0684] The server uses the ThreeDModelGenerator module to generate a 3D model based on the extracted features, which allows the past streetscape and buildings to be reproduced in three dimensions.

[0685] 4.Display:

[0686] The server transmits the generated 3D model to the smart glasses via ARModule and displays it to the user, allowing the user to visually experience the streetscape of the past in a brick-and-mortar store.

[0687] 3. Specific Examples

[0688] For example, when a user visits a historical tourist site, they can wear the smart glasses and experience the townscape of the Edo period, allowing them to enjoy sightseeing while comparing modern scenery with past scenery.

[0689] Example prompt:

[0690] "Please recreate the streets of Asakusa in the Edo period. Analyze old maps and old photographs, generate a 3D model and display it on the smart glasses."

[0691] In this way, users can enjoy shopping and sightseeing while comparing modern-day Asakusa with Asakusa of the Edo period.

[0692] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0693] Step 1:

[0694] The server uses OpenCV to read image data of old maps and old photographs. It receives the image file path of the old map or old photograph as input and obtains image data as output. This image data is used for subsequent analysis processing.

[0695] Step 2:

[0696] The server uses the OldMapAnalyzer module to extract geographic and architectural features from the loaded image data. As input, it receives the image data acquired in step 1, and as output, it generates a dataset containing geographic and architectural features. This dataset will be the basis for reconstructing the streetscape of the past.

[0697] Step 3:

[0698] The server uses the ThreeDModelGenerator module to generate a 3D model based on the extracted features. It receives as input the dataset containing the geographical and architectural features generated in step 2, and generates as output 3D model data. This 3D model data is used to reproduce the appearance of past streets and buildings in three dimensions.

[0699] Step 4:

[0700] The server sends the generated 3D model data to the smart glasses through ARModule. It receives the 3D model data generated in step 3 as input and sends data to be displayed on the smart glasses as output. This data is used for the user to visually experience the past cityscape through the smart glasses.

[0701] Step 5:

[0702] The user wears the smart glasses and experiences the streetscape of the past in a real store. As input, the user receives 3D model data displayed on the smart glasses, and as output, the user visually experiences the streetscape and buildings of the past. Through this experience, the user can enjoy shopping and sightseeing while comparing the modern landscape with the landscape of the past.

[0703] Example 2

[0704] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0705] In order to provide an experience that combines modern streetscapes with past streetscapes, technology is required to acquire appropriate historical data based on the user's current location, and analyze and generate it in real time. However, with conventional technology, there is no system that consistently acquires location information, analyzes historical data, and displays the generated streetscapes, making it difficult for users to smoothly experience past streetscapes.

[0706] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0707] In this invention, the server includes a means for acquiring an old map related to the vicinity of the location information, which is a map from a time before the present, and an old photograph related to the vicinity of the location information, based on the user's location information, a means for extracting geographical features and architectural features from the acquired old map and old photograph, a prompt sentence for instructing to generate a 3D model representing a landscape related to the vicinity of the location information, which is a landscape from a time before the present, based on the extracted geographical features and architectural features, a means for generating the 3D model based on a generative AI model, and a means for displaying an image obtained based on the generated 3D model on smart glasses worn by the user. This allows the user to smoothly enjoy an experience in which modern cityscapes and past cityscapes are blended in real time.

[0708] "Smart glasses" are wearable devices that, when worn by a user, can overlay digital information onto images of the real world.

[0709] “Location Information” means data that describes a User’s current geographic location, such as data obtained using GPS or other location measurement technologies.

[0710] A "server" is a computer system that processes and stores data via a network, and receives, analyzes, and generates information sent from a user's device.

[0711] An "antique map" is a map that shows geographical information from the past, and visually represents historical cityscapes and topography.

[0712] "Old photographs" are photographs taken in times past that record historical buildings and landscapes.

[0713] "Analysis" is the process of examining acquired data in detail and extracting useful information from it.

[0714] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate new data and content, generating past streetscapes based on a prompt text.

[0715] A "prompt" is an instruction entered into a generative AI model to cause it to generate a specific output.

[0716] "Streetscape" refers to the layout and appearance of buildings, roads, and other structures in a particular area.

[0717] "Display" refers to the process of visually presenting digital information to a user by overlaying the information on the smart glasses display.

[0718] This invention is a system that allows a user to wear smart glasses and stand at a specific location, and generates an old streetscape based on information from old maps and old photographs of the location, providing an experience that blends the old streetscape with the real world. A specific embodiment of this system will be described below.

[0719] First, a user puts on the smart glasses and stands at a specific location, for example, in Asakusa, Tokyo. The smart glasses use the built-in GPS sensor to obtain the user's current location information. This location information is then sent to the server in real time.

[0720] The server receives the location information sent from the device. Based on the received location information, it uses geographic information services such as Google Maps API and OpenStreetMap API to obtain old maps and old photographs of the location. The obtained data is analyzed within the server, and geographical and architectural features are abstracted.

[0721] Based on the analyzed data, the server generates a prompt sentence to be input to a generative AI model (e.g., GPT-4 (registered trademark) or DALL-E). For example, the server creates a prompt sentence such as, "Based on geographical features and architectural features, please generate a 3D model that represents the scenery of the Edo period around the current location."

[0722] The generative AI model generates a 3D model representing the old scenery (cityscape) based on the prompt sent from the server. The generated data is returned to the server. The server receives the 3D model data representing the old scenery returned from the generative AI model and sends it to the terminal.

[0723] The device receives 3D model data representing old scenery sent from the server. The images obtained from the received data are superimposed on the real-world image on the smart glasses display, allowing users to enjoy an experience in which the real world and old townscapes are blended in real time.

[0724] As a concrete example, consider the case where a user stands in front of Kaminarimon Gate in Asakusa. When the user wears smart glasses and stands at a specific point in Asakusa, the smart glasses send the location information of that point to the server. The server uses the Google Maps API to obtain old maps and old photos of Asakusa, and inputs the following prompt text to the generative AI model:

[0725]

[0726] The generative AI model generates an image of Asakusa in the Edo period based on this prompt and returns the data to the server. The server then sends the generated data of Asakusa in the Edo period to the device. The device then overlays the received data with real-world images and displays them on the smart glasses display. The user can experience an image that combines modern Asakusa with Asakusa in the Edo period.

[0727] This system allows users to smoothly enjoy an experience in real time where modern cityscapes blend with cityscapes of the past.

[0728] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0729] Step 1:

[0730] A user puts on the smart glasses and stands at a particular point.

[0731] Input: User's location

[0732] Output: Smart glasses get location information

[0733] Specific operation: A user wears the smart glasses and stands at a specific location, for example, in Asakusa, Tokyo. The smart glasses use the built-in GPS sensor to obtain the user's current location information.

[0734] Step 2:

[0735] The device obtains location information and sends it to the server.

[0736] Input: Obtained location information

[0737] Output: Location sent to the server

[0738] How it works: The smart glasses transmit the acquired location information to the server in real time. The location information is sent to the server in the form of latitude and longitude.

[0739] Step 3:

[0740] The server receives the location information and retrieves the corresponding old maps and old photographs.

[0741] Input: Location sent

[0742] Output: Obtained old maps and old photographs

[0743] Specific operation: The server receives the location information sent from the device. Based on the received location information, it uses geographic information services such as Google Maps API and OpenStreetMap API to obtain old maps and old photos of that location.

[0744] Step 4:

[0745] The server analyzes the acquired data and inputs a prompt sentence along with the analysis results into the generative AI model.

[0746] Input: Obtained old maps and old photographs

[0747] Output: Prompt sentence to be input to the generative AI model

[0748] Specific operation: The server analyzes the acquired data of old maps and old photographs. Along with the analysis results, it generates a prompt sentence to be input to a generative AI model (e.g., GPT-4 or DALL-E). For example, it creates a prompt sentence such as, "Based on geographical features and architectural features, please generate a 3D model that represents the scenery of the Edo period around the current location."

[0749] Step 5:

[0750] The generative AI model generates a 3D model representing the old cityscape and returns the data to the server.

[0751] Input: prompt statement

[0752] Output: Generated 3D model data showing the old streetscape

[0753] Specific operation: The generative AI model generates old streetscapes based on the prompts sent from the server. The generated data is returned to the server.

[0754] Step 6:

[0755] The server transmits the generated data to the terminal.

[0756] Input: Generated old streetscape data

[0757] Output: Data sent to the terminal

[0758] Specific operation: The server receives the data of old cityscapes returned by the generative AI model and sends it to the device.

[0759] Step 7:

[0760] The data received by the device is overlaid on images of the real world and displayed.

[0761] Input: Received data of old streetscapes

[0762] Output: Fusion image displayed on the smart glasses display

[0763] Specific operation: The device receives data of old streetscapes sent from the server, and displays the received data on the smart glasses display, overlaying it with images of the real world.

[0764] Step 8:

[0765] Users will experience a fusion of the real world and old-fashioned streetscapes.

[0766] Input: Fusion image displayed on the smart glasses display

[0767] Output: The user's experience

[0768] Specific operation: Through the smart glasses, the user experiences images that combine modern cityscapes with those of the past. For example, the user can enjoy images of modern Asakusa overlapping with those of Asakusa in the Edo period.

[0769] In addition, the user's emotion may be recognized using an emotion engine described later, and the display content of the smart glasses may be adjusted based on the recognized emotion of the user. Specifically, the color tone of the scenery and the facial expressions of people included in the displayed image may be adjusted according to the user's emotion.

[0770] (Application example 2)

[0771] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[0772] Systems that allow users to visually experience a fusion of modern cityscapes and past cityscapes are extremely useful in the fields of tourism and education. However, with current technology, it is difficult for users to obtain, analyze, and visually display historical information about a specific location in real time when they stand there. In addition, technology that uses generative AI models to generate prompt sentences and display past cityscapes based on those sentences is still in its infancy. This poses the challenge of making it difficult for users to experience past cityscapes in real time.

[0773] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0774] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for acquiring information on old maps and old photographs of a specific location when the user stands at the location, a means for generating a prompt sentence using a generative AI model based on the acquired information, and a means for visually displaying the old streetscape based on the generated prompt sentence. This enables the user to experience the past streetscape of a specific location in real time simply by standing at the location.

[0775] "Smart glasses" are wearable devices that allow users to overlay digital information onto the real world.

[0776] An "antique map" is a map that shows geographical information from the past, recording historical topography and the layout of buildings.

[0777] "Old photographs" are photographs of landscapes, buildings, and people from the past, and serve as visual records of historical information.

[0778] "Means of analysis" refers to techniques and methods for processing the information from acquired old maps and old photographs as data and extracting geographical and architectural features.

[0779] "Means of generation" refers to the techniques and methods for recreating past streetscapes and landscapes as digital data based on analyzed information.

[0780] "Display means" refers to the technology or method for displaying the generated past cityscapes and scenery on the smart glasses.

[0781] A "particular location" is a location where a user is standing and about which historical information relating to that location is retrieved.

[0782] A "generative AI model" is a model that uses artificial intelligence to analyze data and generate output for a specific purpose.

[0783] A "prompt sentence" is an instruction sentence output by a generative AI model, and is text generated based on specific information.

[0784] "Means for visual display" refers to techniques or methods for displaying the generated past streetscapes or scenery so that the user can visually experience them.

[0785] The system for implementing the present invention includes smart glasses, a server, and a generative AI model. A specific embodiment of the system is described below.

[0786] First, when a user wears the smart glasses and stands at a specific location, the smart glasses send the location information of that location to the server. Based on the location information, the server retrieves information on old maps and old photographs related to that location from a database that stores past geographical information and historical photographs.

[0787] Next, the server analyzes the information on the acquired old maps and old photographs. Image processing techniques are used for the analysis to extract geographical and architectural features. Specifically, image processing libraries such as OpenCV are used to extract feature points from maps and photographs.

[0788] Based on the analyzed information, the server generates a prompt sentence using a generative AI model. For the generative AI model, a natural language generation model such as GPT-3 is used. The generated prompt sentence is an instruction sentence for reproducing the past streetscape.

[0789] Based on the generated prompt, the server generates digital data of the streetscape of the past. This generation is done using 3D modeling software and rendering engines, such as Blender and Unity.

[0790] The generated past streetscapes are sent to the smart glasses and displayed in the user's field of view, allowing the user to have a visual experience that blends the real world with the past streetscapes.

[0791] As a concrete example, when a user stands at Tokyo Station, the server retrieves old photos of Tokyo Station, analyzes them, and generates a prompt. The generated prompt is "Please show me the old streetscape of Tokyo Station." Based on this prompt, the server generates an old streetscape of Tokyo Station and displays it on the smart glasses. The user can enjoy walking around while comparing the modern Tokyo Station with the old streetscape of Tokyo Station.

[0792] In this way, the present invention enables a user to experience a past streetscape of a particular location in real time, simply by standing at that location.

[0793] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0794] Step 1:

[0795] A user puts on the smart glasses and stands at a particular point.

[0796] Input: User's location

[0797] Output: Location information is sent from the smart glasses to the server.

[0798] Specific operation: The GPS function of the smart glasses obtains the user's current location and sends that location information to the server.

[0799] Step 2:

[0800] Based on the location information, the server retrieves information on old maps and old photographs related to that location from a database.

[0801] Input:Location

[0802] Output: Information on old maps and old photographs

[0803] Specific operation: The server queries the database using the location information as a key and retrieves the relevant data on old maps and old photographs.

[0804] Step 3:

[0805] The server analyzes the information on old maps and old photographs it has acquired.

[0806] Input: Information on old maps and old photographs

[0807] Output: Geographical and architectural features

[0808] Specific operation: The server uses image processing libraries such as OpenCV to extract feature points from old maps and old photographs.

[0809] Step 4:

[0810] The server generates a prompt sentence based on the parsed information using a generative AI model.

[0811] Input: Geographical and architectural features

[0812] Output: Prompt statement

[0813] Specific operation: The server inputs feature data into a generative AI model such as GPT-3 and generates prompt sentences to recreate past cityscapes.

[0814] Step 5:

[0815] The server generates past streetscapes as digital data based on the generated prompt sentence.

[0816] Input: Prompt statement

[0817] Output: Digital data of past streetscapes

[0818] Specific operation: The server uses 3D modeling software such as Blender or Unity to generate a 3D model based on the prompt text.

[0819] Step 6:

[0820] The server transmits the generated past streetscapes to the smart glasses and displays them in the user's field of view.

[0821] Input: Digital data of past streetscapes

[0822] Output: A past cityscape displayed on smart glasses

[0823] Specific operation: The server sends the generated 3D model to the smart glasses, which overlay it on the user's field of view.

[0824] In this way, users can have a visual experience that blends the real world with past cityscapes.

[0825] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0826] "Example 1"

[0827] The present invention is a system that works in conjunction with smart glasses, analyzes information from old maps and old photographs, generates old streetscapes based on the analyzed information, and displays the generated streetscapes on the smart glasses. It further includes an emotion engine that recognizes the user's emotions. This emotion engine estimates the user's emotions from the user's facial expressions, tone of voice, and operation patterns of the smart glasses. Based on the estimated emotions, the system adjusts the color tones and music of the generated streetscapes, the facial expressions of people displayed, and the like, to provide an experience that matches the user's emotions.

[0828] "Example 2"

[0829] As a specific example, when a user puts on smart glasses and stands at a specific location, information about old maps and old photographs of that location is sent to the system. An analysis module in the system extracts geographical and architectural features from the sent information, and generates old streetscapes based on the extracted features. The generated streetscape is displayed on the smart glasses, and the user experiences a fusion of the real world and old streetscapes. At this time, the emotion engine detects the user's joy, surprise,

[0830] The system recognizes emotions such as nostalgia and adjusts the display of the streetscape according to those emotions. For example, when the user is feeling happy, the colors of the streetscape are made vivid and the people's expressions brighter. On the other hand, when the user is feeling nostalgic, the colors of the streetscape are made subdued and the people's expressions calmer.

[0831] The process flow of each embodiment will be described below.

[0832] "Example 1"

[0833] Step 1: A user puts on the smart glasses and stands at a specific point.

[0834] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[0835] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[0836] Step 4: Generate old streetscapes based on the extracted features.

[0837] Step 5: Display the generated cityscape on the smart glasses.

[0838] Step 6: The emotion engine recognizes the user's emotion and adjusts the display of the cityscape according to that emotion.

[0839] "Example 2"

[0840] Step 1: A user puts on the smart glasses and stands at a specific point.

[0841] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[0842] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[0843] Step 4: Generate old streetscapes based on the extracted features.

[0844] Step 5: Display the generated cityscape on the smart glasses.

[0845] Step 6: The emotion engine recognizes the user's emotion and adjusts the display of the streetscape according to that emotion. For example, when the user is feeling happy, the colors of the streetscape are made vivid and the people's expressions brighter. On the other hand, when the user is feeling nostalgic, the colors of the streetscape are made muted and the people's expressions calmer.

[0846] Example 1

[0847] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0848] With conventional technology, it was difficult to analyze information from old maps and old photographs and recreate old streetscapes. In addition, it was not possible to adjust the display content according to the user's emotions, making it impossible to provide a consistent experience to the user. Furthermore, there was a lack of technology to change the display content in real time in conjunction with smart glasses. This led to problems such as a decrease in the sense of immersion and satisfaction when users experienced historical streetscapes.

[0849] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0850] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for recognizing a user's emotion, and a means for adjusting the display content based on the recognized emotion. This allows a user to experience old streetscapes in real time based on information on old maps and old photographs, and further allows the experience to be adjusted according to the user's emotion.

[0851] "Smart glasses" are wearable devices that can display visual information as augmented reality when worn by a user.

[0852] An "antique map" is a map that shows geographical information from the past, including historical geographical features and the layout of buildings.

[0853] "Old photographs" are photographs taken in the past that record historical buildings and landscapes.

[0854] An "analysis module" is software or hardware used to analyze information from old maps and old photographs and extract geographical and architectural features.

[0855] A "3D model" is a three-dimensional digital model that is generated based on analyzed geographical features and architectural characteristics.

[0856] An "emotion engine" is software or hardware that recognizes and estimates a user's emotions from the user's facial expressions, tone of voice, operation patterns, etc.

[0857] The "means for adjusting the display content" refers to software or hardware for changing the color tones of the displayed streetscape, the music, the facial expressions of people displayed, etc., based on the recognized emotions of the user.

[0858] "Geographic features" refers to geographic elements such as rivers, roads, and mountains on a map or photograph.

[0859] "Building features" refers to elements such as the shape, layout, and structure of a building on a map or photograph.

[0860] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0861] This invention is a system that works in conjunction with smart glasses, analyzes information from old maps and old photographs, generates old streetscapes based on the analyzed information, and displays the generated streetscapes on the smart glasses. It also includes a function to recognize the user's emotions and adjust the display content based on the emotions.

[0862] Hardware and Software Used

[0863] The server uses the following hardware and software:

[0864] Image analysis library: OpenCV, TensorFlow

[0865] 3D modeling software: Unity, Unreal Engine

[0866] Emotion Recognition API: Microsoft Azure Emotion API, Google Cloud Speech-to-Text API

[0867] Communication technology: Wi-Fi, Bluetooth

[0868] Data processing and calculation

[0869] The server receives data of old maps and old photographs uploaded by users. The received data is analyzed using image analysis libraries (OpenCV and TensorFlow). The analysis module extracts geographical features (rivers, roads, mountains, etc.) and architectural features (shape, layout, structure, etc. of buildings).

[0870] Based on the analysis results, the server uses 3D modeling software (Unity or Unreal Engine) to generate a 3D model. The generated 3D model reproduces the old streetscape. For example, when recreating a streetscape from the Edo period, the layout of the buildings and roads at that time is faithfully reproduced.

[0871] The generated 3D model is then sent to the smart glasses via Wi-Fi or Bluetooth, where the smart glasses display the received 3D model, giving the user the experience of traveling back in time.

[0872] Emotion recognition and display content adjustment

[0873] The server recognizes the user's emotions using an emotion engine. The emotion engine estimates the user's emotions from the user's facial expressions, tone of voice, and operation patterns of the smart glasses. For example, it uses Microsoft Azure's Emotion API or Google Cloud's Speech-to-Text API.

[0874] After the emotion engine estimates the user's emotion, the server adjusts the display content based on that emotion. For example, if the server estimates that the user is having fun, it will brighten the colors of the streetscape and play cheerful music. It will also adjust the facial expressions of people displayed to smile.

[0875] Examples of concrete examples and prompts

[0876] For example, if a user uploads old maps and old photographs from the Edo period, the server receives this data and uses an analysis module to extract geographical and architectural features from the Edo period. Based on the analysis results, a 3D model of the Edo period streetscape is generated using Unity. The generated 3D model is sent to the smart glasses via Wi-Fi, and the user can experience the Edo period streetscape through the smart glasses.

[0877] When a user is having fun walking through a street, the emotion engine infers from the user's facial expression that the user is having fun. The server brightens the colors of the street and plays cheerful music. The server also adjusts the facial expressions of the people displayed to smile.

[0878] Example of a prompt to be input to a generative AI model:

[0879] "Analyze old maps and photographs from the Edo period and generate a 3D model of the streets from that period. When the user is enjoying the game, brighten the colors of the streets and play fun music."

[0880] In this way, a system is realized in which the server, terminal, and user work together to provide the user with an experience of historical townscapes that correspond to their emotions.

[0881] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0882] Step 1:

[0883] Users upload old maps and old photographs.

[0884] A user uploads data of old maps and old photographs using a terminal (e.g., a smartphone or tablet) connected to the smart glasses. As input, they provide image files in JPEG or PNG format. As output, the uploaded data is sent to the server.

[0885] Step 2:

[0886] The server receives the uploaded data.

[0887] The server receives data of old maps and old photographs uploaded by users. As input, it receives image files sent by users. As output, it stores the received data in a database in the server.

[0888] Step 3:

[0889] The server analyzes the data using an analysis module.

[0890] The server starts the analysis module and analyzes the received data of old maps and old photographs. It uses the stored image files as input. The analysis module uses image analysis libraries such as OpenCV and TensorFlow to extract geographical and architectural features. The extracted feature information is obtained as output.

[0891] Step 4:

[0892] The server generates a 3D model based on the analysis results.

[0893] The server uses 3D modeling software such as Unity or Unreal Engine to generate a 3D model based on the feature information obtained from the analysis module. It uses the extracted feature information as input. As output, a 3D model that reproduces the old streetscape is generated.

[0894] Step 5:

[0895] The server transmits the generated 3D model to the smart glasses.

[0896] The server sends the generated 3D model to the smart glasses. As input, the server uses the generated 3D model. As output, the 3D model is sent to the smart glasses. For transmission, wireless communication technologies such as Wi-Fi and Bluetooth are used.

[0897] Step 6:

[0898] Smart glasses display 3D models.

[0899] The smart glasses display the 3D model received from the server. As input, they use the 3D model sent from the server. As output, the user can visually experience the old streetscape as if they had traveled back in time.

[0900] Step 7:

[0901] Users experience the cityscape through smart glasses.

[0902] A user can wear the smart glasses and walk around the displayed 3D model of the cityscape. As input, we use the 3D model displayed on the smart glasses. As output, the user experiences the historical cityscape.

[0903] Step 8:

[0904] The server uses an emotion engine to recognize the user's emotion.

[0905] The server starts an emotion engine and recognizes the user's emotions from the user's facial expressions, tone of voice, and operation patterns of the smart glasses. The inputs used are the user's facial expression data, voice data, and operation data. The emotion engine uses Microsoft Azure's Emotion API and Google Cloud's Speech-to-Text API. The output is an estimated user's emotions.

[0906] Step 9:

[0907] The server adjusts the display content based on the user's emotions.

[0908] The server adjusts the display content of the generated 3D model based on the user's emotions estimated by the emotion engine. As input, it uses the estimated user's emotion data. As output, the color tones of the streetscape, music, and facial expressions of people displayed are adjusted. For example, if it is estimated that the user is having fun, the color tones of the streetscape are brightened and cheerful music is played. The facial expressions of people displayed are also adjusted to smile.

[0909] (Application example 1)

[0910] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0911] Conventional smart glasses-based systems can analyze old maps and old photographs to recreate old streetscapes, but they cannot adjust the display content according to the user's emotions, so the user experience is uniform and it is difficult to provide an optimized experience for each individual user. In addition, there is a lack of concrete measures to improve the shopping experience in physical stores. This makes it difficult to increase user satisfaction and stimulate purchasing desire in physical stores.

[0912] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0913] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for recognizing the user's emotions, and a means for adjusting the display content based on the recognized emotions. This makes it possible to provide optimal display content according to the user's emotions, improving the shopping experience in a brick-and-mortar store and increasing the user's satisfaction and purchasing motivation.

[0914] "Smart glasses" are wearable devices that can display visual information as augmented reality when worn by the user.

[0915] An "antique map" is a map that shows geographical information from the past, recording the locations of historical geographical features and buildings.

[0916] "Old photographs" are photographs taken in the past that record historical buildings and landscapes.

[0917] "Means of analysis" refer to the technical means for analyzing information from old maps and old photographs as digital data and extracting geographical and architectural features.

[0918] The "means of generation" refers to the technical means for generating a three-dimensional model of a past streetscape based on the analyzed information.

[0919] "Means for displaying" means the technical means for displaying the generated 3D model on the smart glasses and providing visual information to the user.

[0920] "Means of recognizing emotions" refers to technical means for inferring emotions from a user's facial expressions, tone of voice, operation patterns, etc.

[0921] "Means for adjusting the display content" refers to technical means for adjusting the color tones of the displayed cityscape, the music, the facial expressions of people displayed, etc., based on the recognized emotions of the user.

[0922] The system for implementing this invention uses hardware such as smart glasses, a server, a camera, a microphone, etc., and software such as OpenCV, TensorFlow, Affectiva SDK, Unity, FMOD, etc. The specific configuration and operation of the system are described below.

[0923] System Configuration

[0924] 1. Smart glasses:

[0925] It is a wearable device that is worn by the user and displays visual information as augmented reality.

[0926] 2. Server:

[0927] It is a central processing device that analyzes information from old maps and old photographs and generates old cityscapes based on the analyzed information.

[0928] 3. Camera:

[0929] A device for capturing a user's facial expressions.

[0930] 4. Microphone:

[0931] A device for capturing the tone of a user's voice.

[0932] Software configuration

[0933] 1. OpenCV:

[0934] This is a library for analyzing images of old maps and old photographs.

[0935] 2. TensorFlow:

[0936] A framework for running machine learning models for image analysis.

[0937] 3. Affectiva SDK:

[0938] It is a software development kit for recognizing user emotions.

[0939] 4. Unity:

[0940] It is a game engine for generating 3D models and adjusting the display content.

[0941] 5. FMOD:

[0942] An audio engine for controlling music and sound effects.

[0943] System Operation

[0944] 1. Analysis of old maps and photographs:

[0945] The server uses OpenCV and TensorFlow to analyze old maps and photographs and extract geographical and architectural features.

[0946] 2. Generate 3D model:

[0947] The server uses Unity to generate a 3D model of the old townscape based on the extracted features.

[0948] 3. Emotion Recognition:

[0949] The server analyzes the data obtained from the camera and microphone using the Affectiva SDK to recognize the user's emotions.

[0950] 4. Adjust the display:

[0951] The server uses Unity and FMOD to adjust the content displayed based on the emotions it recognizes, such as changing the color tones of the cityscape, the music, and the facial expressions of people displayed.

[0952] Examples

[0953] If a user wants to experience the streets of the Edo period, they can wear the smart glasses and walk around a brick-and-mortar store. The streets of the Edo period, recreated from old maps and photographs, are displayed on the smart glasses, giving the user the experience of traveling back in time to that era. Furthermore, if the user has a happy expression, the colors of the streets become brighter and cheerful music plays.

[0954] Examples of prompt statements

[0955] "Develop an application for smart glasses that recreates the streets of the Edo period and adjusts the display content according to the user's emotions. Analyze old maps and old photographs, generate 3D models, and use an emotion recognition engine to adjust the display content."

[0956] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0957] Step 1:

[0958] The server receives image data of old maps and old photographs. As input, image files of old maps and old photographs are provided. The server reads these images using OpenCV and performs preprocessing. Specifically, it removes noise from the images and adjusts the resolution. As output, it obtains the preprocessed image data.

[0959] Step 2:

[0960] The server inputs the preprocessed image data into a TensorFlow model to extract geographic and architectural features. The preprocessed image data is provided as input. The server uses TensorFlow to perform image analysis and extract features. Specifically, it identifies building outlines and topographical features. The extracted feature data is obtained as output.

[0961] Step 3:

[0962] The server uses Unity to generate a 3D model based on the extracted feature data. The feature data is provided as input. The server executes Unity scripts to generate a 3D model that reproduces the past streetscape. Specifically, it reproduces the layout of buildings and the terrain. The generated 3D model data is obtained as output.

[0963] Step 4:

[0964] The user wears the smart glasses and transmits real-time data to the server through the camera and microphone. As input, the user's facial expression data and voice data are provided. The server uses the Affectiva SDK to analyze these data and recognize the user's emotions. Specifically, it analyzes changes in facial expressions and tone of voice. As output, it obtains the recognized emotion data.

[0965] Step 5:

[0966] The server uses Unity and FMOD to adjust the display content based on the recognized emotion data. Emotion data and 3D model data are provided as input. The server executes Unity scripts to adjust the color tone of the cityscape, the music, and the facial expressions of the people displayed. Specifically, it brightens the color tone and changes the music depending on the emotion. The adjusted display content data is obtained as output.

[0967] Step 6:

[0968] The server transmits the adjusted display content data to the smart glasses to display it to the user. As an input, the adjusted display content data is provided. The server transmits data to the display of the smart glasses to provide visual information to the user. Specifically, the reproduced cityscape and the adjusted display content are displayed on the smart glasses. As an output, visual information experienced by the user is obtained.

[0969] Example 2

[0970] Next, a description will be given of Example 2 of the embodiment 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0971] In conventional systems, it was difficult for a user to visually reproduce the past streetscape of a particular location when the user stood at that location. In addition, the display content could not be dynamically adjusted according to the user's emotions, limiting the user experience. This led to the issue that users could not experience the past streetscape more realistically.

[0972] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0973] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, and a means for recognizing the user's emotions and adjusting the display according to the emotions. This makes it possible to visually reproduce the past streetscape of a specific location when the user stands at that location in real time, and further dynamically adjust the display content according to the user's emotions.

[0974] "Smart glasses" are wearable devices that, when worn by a user, can overlay digital information onto visual information from the real world.

[0975] An "antique map" is a map that shows geographical information from the past, and includes historical geographical information for specific locations or areas.

[0976] An "archive photograph" is a photograph taken in the past that contains historical visual information of a particular location or area.

[0977] "Means of analysis" refers to the technical means for receiving information from old maps and old photographs and extracting geographical and architectural features from that information.

[0978] The "means of generation" refers to the technical means for recreating the streetscape of the past based on the analyzed information, specifically the generation of a three-dimensional model.

[0979] The "display means" refers to the technical means for displaying the generated past streetscape on the smart glasses.

[0980] "Means for recognizing emotions" refers to a technical means for detecting emotions from a user's facial expressions, voice, etc., and adjusting the display content based on those emotions.

[0981] This invention is a system in which a user wears smart glasses and stands at a specific location, which visually recreates the past streetscape of that location, and dynamically adjusts the display content according to the user's emotions.

[0982] The server receives information on old maps and old photos sent from the smart glasses when a user stands at a specific location. The received information is passed to the analysis module, which extracts geographical and architectural features. The analysis module generates old streetscapes based on the extracted features. A generative AI model is used for this generation. The generated streetscape data is then sent to the smart glasses.

[0983] When a user stands at a specific spot, the device (smart glasses) sends information about old maps and old photographs of that spot to the server. The device receives data on old streetscapes generated by the server and displays it blended with the real world. The device also uses an emotion engine to recognize the user's emotions and adjusts the display according to those emotions. For example, if the user is feeling happy, the colors of the streetscape will be made more vivid and the people's expressions will be brighter. Conversely, if the user is feeling nostalgic, the colors of the streetscape will be made more subdued and the people's expressions will be made calmer.

[0984] The user puts on the smart glasses and stands at a specific location. The glasses send information about old maps and old photos of that location to the server, and receive the old streetscape generated by the server. The user can enjoy an experience where the real world and the old streetscape are fused together. In addition, the user's emotions are recognized and the display is adjusted according to those emotions, providing a more moving experience.

[0985] As a concrete example, let's say a user is standing in Asakusa, Tokyo. When the user wears smart glasses and stands in front of Kaminarimon Gate in Asakusa, the smart glasses send information about old maps and old photographs of that location to the server. The server analyzes the received information and generates the streetscape of Asakusa in the Edo period. The generated streetscape data is sent to the smart glasses as a 3D model. The smart glasses display the Edo period streetscape superimposed on the real scenery in front of Kaminarimon Gate. If the user is feeling happy, the colors of the Edo period streetscape will become more vivid and people's expressions will become brighter. On the other hand, if the user is feeling nostalgic, the colors of the streetscape will become more subdued and people's expressions will become calmer.

[0986] Examples of prompts to be input to a generative AI model include the following:

[0987] "When a user stands in Asakusa, Tokyo, generate an image of the streets of Asakusa from the Edo period and display it on the smart glasses. Adjust the display according to the user's emotions. If the user is feeling joy, make the colors vivid and brighten the people's expressions. If the user is feeling nostalgia, make the colors calmer and calm the people's expressions."

[0988] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0989] Step 1:

[0990] A user puts on the smart glasses and stands at a specific point.

[0991] Specific operation: The user wears the smart glasses and stands in front of the Kaminarimon Gate in Asakusa, Tokyo, for example. The smart glasses acquire the user's location information.

[0992] Input: User's location

[0993] Output: Get location information

[0994] Step 2:

[0995] The terminal (smart glasses) sends information about old maps and old photographs of that location to the server.

[0996] How it works: The smart glasses use GPS and Wi-Fi location information to determine the user's current location and send that information to a server.

[0997] Input: User's location

[0998] Output: Send location information

[0999] Step 3:

[1000] The server passes the received information to an analysis module, which extracts geographical and architectural features.

[1001] Specific operation: The server receives information on old maps and old photographs sent from the terminal and passes it to the analysis module. The analysis module extracts geographical features (e.g., road layout and river location) and architectural features (e.g., building shape and height).

[1002] Input: Information on old maps and old photographs

[1003] Output: Extraction of geographic and architectural features

[1004] Step 4:

[1005] The server generates old streetscapes based on the extracted features.

[1006] Specific operation: Based on the features extracted by the analysis module, the server uses a generative AI model to generate old streetscapes. For example, it recreates the streetscape of Asakusa in the Edo period.

[1007] Input: Geographical and architectural features

[1008] Output: Generation of old streetscape

[1009] Step 5:

[1010] The server transmits the generated streetscape data to the terminal.

[1011] Specific operation: The server sends the generated data of the old cityscape to the terminal as a 3D model or image data.

[1012] Input: Data of old streetscape

[1013] Output: Sending streetscape data

[1014] Step 6:

[1015] The streetscape data received by the device is blended with the real world and displayed.

[1016] Specific operation: The device overlays the received streetscape data onto the real-world scenery, and displays the fusion of real-world scenery and the old streetscape on the smart glasses display.

[1017] Input: Streetscape data

[1018] Output: Real-world blended display

[1019] Step 7:

[1020] The device recognizes the user's emotions using an emotion engine and adjusts the display accordingly.

[1021] Specific operation: The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, if the user is feeling happy, the colors of the streetscape will be brighter and the people's expressions will be brighter. On the other hand, if the user is feeling nostalgic, the colors of the streetscape will be muted and the people's expressions will be calmer.

[1022] Input: User emotion data

[1023] Output: Adjusting display according to emotions

[1024] (Application example 2)

[1025] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the smart glasses 214 will be referred to as a "terminal".

[1026] In modern brick-and-mortar stores, there is a lack of ways to allow customers to visually experience past history and culture. In addition, technology to dynamically adjust display content according to customer emotions is also underdeveloped. This makes it difficult to increase customer interest and satisfaction.

[1027] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1028] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for recognizing a user's emotion, and a means for adjusting the display content according to the recognized emotion. This allows customers to visually experience past history and culture in a brick-and-mortar store, and the experience can be dynamically adjusted according to the customer's emotion.

[1029] "Smart glasses" are glasses-type devices that users can wear to experience augmented reality (AR) and virtual reality (VR).

[1030] An "antique map" is a map that shows geographical information from the past, and records historical cityscapes and topography.

[1031] "Old photographs" are photographs taken in the past that record historical buildings and landscapes.

[1032] "Means of analysis" refers to the technology and equipment required to process information from old maps and old photographs as digital data and to understand and analyze their contents.

[1033] "Means of generation" refers to technology and devices for digitally recreating past cityscapes and landscapes based on analyzed information.

[1034] "Display means" refers to the technology and devices for projecting the generated past cityscapes and scenery onto the display of smart glasses.

[1035] "Means of recognizing emotions" refers to technology or devices for detecting and analyzing emotions from a user's facial expressions, voice, etc.

[1036] "Means for adjusting the display content" refers to technology or devices for dynamically changing the color tone, brightness, etc. of the displayed past cityscapes or scenery in response to the recognized emotions of the user.

[1037] A system for implementing the present invention includes smart glasses, a server, and various software modules. Specific embodiments are described below.

[1038] System Configuration

[1039] The system includes smart glasses, a server, and the following software modules:

[1040] Image processing library (e.g. OpenCV)

[1041] Emotion recognition library (e.g. EmotionRecognizer)

[1042] Augmented reality display library (e.g. ARDisplay)

[1043] Custom modules for obtaining old maps and old photographs (e.g. historical_data)

[1044] Hardware

[1045] Smart glasses (e.g. Microsoft HoloLens)

[1046] Smart glasses with built-in camera

[1047] software

[1048] OpenCV: Image processing library

[1049] EmotionRecognizer: Emotion recognition library

[1050] ARDisplay: an augmented reality display library

[1051] historical_data: A custom module for retrieving historical maps and photographs.

[1052] Processing flow

[1053] 1. Acquire video from the smart glasses camera

[1054] The camera built into the smart glasses captures footage in real time.

[1055] 2. Get current location information

[1056] The GPS function of the smart glasses is used to obtain the user's current location.

[1057] 3. Obtain information on old maps and old photographs

[1058] Based on the acquired location information, the server retrieves corresponding data on old maps and old photographs from the historical_data module.

[1059] 4. Emotion recognition

[1060] The server uses the video captured by the smart glasses' camera to recognize the user's emotions using the EmotionRecognizer library.

[1061] 5. Adjusting the display content

[1062] Depending on the recognized emotion, the ARDisplay library is used to adjust the color tone and brightness of the displayed content (old maps and old photographs).

[1063] 6. AR display

[1064] Adjusted old maps and old photographs are overlaid on the display of the smart glasses.

[1065] Examples

[1066] When a user enters a traditional Japanese sweets shop, the smart glasses will show the shop's original appearance and manufacturing process from the past. If the user is feeling happy, the images displayed will be bright and vivid. If the user is feeling nostalgic, the images will be muted.

[1067] Examples of prompt statements

[1068] "When a user puts on the smart glasses and enters a long-established Japanese confectionery shop, please develop an application that displays the shop's history and the manufacturing process from the past. Please also add a function to adjust the display content according to the user's emotions."

[1069] The above is a specific embodiment for carrying out the present invention.

[1070] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1071] Step 1:

[1072] The server acquires video in real time from the smart glasses' camera.

[1073] Input: Video data from the camera built into the smart glasses

[1074] Output: Real-time video frames

[1075] Specific operation: The camera in the smart glasses captures the user's field of vision and sends the video data to the server.

[1076] Step 2:

[1077] The server obtains the user's current location information using the smart glasses' GPS function.

[1078] Input: GPS data from smart glasses

[1079] Output: Current location information (latitude and longitude)

[1080] Specific operation: The GPS sensor in the smart glasses obtains the current location information and sends the data to the server.

[1081] Step 3:

[1082] Based on the acquired location information, the server retrieves corresponding data on old maps and old photographs from the historical_data module.

[1083] Input: Current location

[1084] Output: Data of old maps and old photographs

[1085] Specific operation: The server sends a query to the historical_data module based on the location information and retrieves the relevant data on old maps and old photographs.

[1086] Step 4:

[1087] The server uses the video captured by the smart glasses' camera to recognize the user's emotions using the EmotionRecognizer library.

[1088] Input: Real-time video frames

[1089] Output: User emotion data (e.g., happiness, nostalgia)

[1090] Specific operation: The server inputs video frames into the EmotionRecognizer library and analyzes emotions from the user's facial expressions and voice.

[1091] Step 5:

[1092] The server uses the ARDisplay library to adjust the color and brightness of the displayed content (old maps and old photographs) according to the recognized emotion.

[1093] Input: User emotion data, data on old maps and old photographs

[1094] Output: Adjusted data of historical maps and photographs

[1095] Specific operation: Based on the emotion data, the server dynamically changes the color tone and brightness of old maps and old photographs using the ARDisplay library.

[1096] Step 6:

[1097] The server then overlays the adjusted old maps and old photographs onto the smart glasses' display.

[1098] Input: Adjusted historical map and photo data, real-time video frames

[1099] Output: Augmented reality video displayed on smart glasses

[1100] Specific operation: The server overlays the adjusted old maps and old photos onto real-time video and sends them to the smart glasses display for display.

[1101] The above are the specific process steps for carrying out the present invention.

[1102] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1103] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1104] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1105] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1106] [Third embodiment]

[1107] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1108] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1109] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1110] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1111] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1112] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1113] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1114] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1115] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1116] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1117] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1118] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1119] "Example 1"

[1120] As one embodiment of the present invention, a system that works with smart glasses is provided. The system includes a module that analyzes information on old maps and old photographs. The analysis module extracts geographical features and architectural features from the old maps and old photographs. A module that generates old streetscapes operates based on the extracted features. The generation module generates a 3D model based on the extracted geographical features and architectural features. The generated streetscape is displayed on the smart glasses, allowing the user to experience a time-traveling experience.

[1121] "Example 2"

[1122] For example, when a user wears smart glasses and stands at a specific location, information about old maps and old photographs of that location is sent to the system. The system analyzes the information and generates an old streetscape of that location. The generated streetscape is displayed on the smart glasses, allowing the user to experience a fusion of the real world and the old streetscape.

[1123] The process flow of each embodiment will be described below.

[1124] "Example 1"

[1125] Step 1: A user puts on the smart glasses and stands at a specific point.

[1126] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[1127] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[1128] Step 4: The module that generates old streetscapes runs based on the extracted features.

[1129] Step 5: The generated streetscape is displayed on the smart glasses, giving the user an experience that blends the real world with the old streetscape.

[1130] "Example 2"

[1131] Step 1: A user puts on the smart glasses and stands at a specific point.

[1132] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[1133] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[1134] Step 4: The module that generates old streetscapes runs based on the extracted features.

[1135] Step 5: The generated streetscape is displayed on the smart glasses, giving the user an experience that blends the real world with the old streetscape.

[1136] Example 1

[1137] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1138] With conventional technology, it was difficult to reproduce streetscapes of the past using old maps and old photographs. In addition, the means by which users could experience streetscapes of the past were limited. This restricted the use of the technology in fields such as the reproduction of historical streetscapes, education, and tourism.

[1139] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1140] In this invention, the server includes a means for a user to upload old maps and old photographs from a terminal, a means for the server to receive the uploaded data, a means for the server to analyze the data using an analysis module and extract geographical features and architectural features, a means for the server to generate a 3D model using a generation module based on the features extracted by the server, a means for the server to transmit the generated 3D model to the smart glasses, and a means for the smart glasses to display the 3D model and for the user to experience it. This allows the user to experience past streetscapes in real time.

[1141] A "user" is an individual or group that uses the system to upload old maps and old photographs and experience streetscapes of the past.

[1142] A "terminal" is a device that a user uses to upload old maps and old photographs, and includes smartphones, personal computers, etc.

[1143] "Server" refers to a computer system that receives data uploaded by users and performs analysis and generation of 3D models.

[1144] The "analysis module" is a software component that analyzes data from old maps and old photographs and extracts geographical and architectural features.

[1145] "Geographical features" are geographical elements such as rivers, roads, and mountains that are included in old maps and old photographs.

[1146] "Building characteristics" refers to information such as the shape and layout of buildings contained in old maps and photographs.

[1147] The "generative module" is a software component for generating 3D models based on extracted geographical and architectural features.

[1148] A "3D model" is a three-dimensional digital representation generated based on extracted features.

[1149] "Smart glasses" are a wearable device that allows users to visually experience three-dimensional models.

[1150] "Uploading" refers to the act of a user using a terminal to send data of old maps or old photographs to a server.

[1151] "Receiving" is the act of the server taking in data sent by the user.

[1152] "Transmitting" refers to the act of the server sending the generated 3D model to the smart glasses.

[1153] "Display" refers to the act of the smart glasses projecting the received 3D model into the user's field of vision.

[1154] An "experience" is an act in which a user can visually sense a streetscape of the past through the smart glasses.

[1155] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1156] This invention is a system that allows users to experience past streetscapes using old maps and old photographs. This system includes a series of processes in which the user uploads old maps and old photographs from a terminal, the server analyzes the data, generates a 3D model, and displays it to the user through smart glasses.

[1157] Hardware and Software Used

[1158] Terminal

[1159] Users upload old maps and old photographs using devices such as smartphones and computers, allowing them to easily access the system and provide data.

[1160] server

[1161] The server receives data uploaded by users, analyzes it, and generates 3D models. The following software is installed on the server:

[1162] Analysis module: Uses Python's OpenCV library and machine learning frameworks such as TensorFlow to analyze data from old maps and photographs and extract geographic and architectural features.

[1163] Generation module: Using 3D modeling software such as Blender or Unity, a 3D model is generated based on the extracted features.

[1164] Smart Glasses

[1165] The smart glasses display the 3D model sent from the server in the user's field of vision, allowing the user to experience past streetscapes in real time.

[1166] Examples

[1167] For example, if a user uploads an old map of Tokyo in the Edo period in JPEG format, the server receives this data and uses OpenCV to detect the outlines of rivers, roads, and buildings.The server then uses Blender to generate a 3D model of the streets of Tokyo in the Edo period based on the detected features.This 3D model is sent to the smart glasses via Wi-Fi, allowing the user to experience Tokyo in the Edo period through the smart glasses.

[1168] Examples of prompt statements

[1169] Examples of prompts to be input to a generative AI model include the following:

[1170] "Analyze old maps of Tokyo from the Edo period and generate a 3D model of the cityscape from that time."

[1171] Using this prompt, the generative AI model can perform the necessary analysis and generation processes to provide the user with an experience that recreates a historical cityscape.

[1172] In this way, the present invention provides a system that allows a user to experience past streetscapes in real time.

[1173] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1174] Program processing flow

[1175] Step 1: User uploads old maps and old photos from their device

[1176] Users upload data on old maps and old photographs to the system using devices such as smartphones or computers. For example, an old map of Tokyo from the Edo period can be uploaded in JPEG format.

[1177] Input: Data of old maps and old photographs (e.g. JPEG format images)

[1178] Output: Uploaded data is sent to the server

[1179] Step 2: The server receives the uploaded data

[1180] The server receives data of old maps and old photographs uploaded by users. This data is stored in a specific directory on the server.

[1181] Input: Data on old maps and old photographs sent by users

[1182] Output: Data stored in a specific directory on the server

[1183] Step 3: The server analyzes the data using the analysis module to extract geographic and architectural features.

[1184] The server uses machine learning frameworks such as Python's OpenCV library and TensorFlow to analyze the uploaded data, for example to detect the outlines of rivers, roads, and buildings in old maps and extract their features.

[1185] Input: Data of old maps and old photographs stored on the server

[1186] Output: Extracted geographic and architectural features (e.g. rivers, roads, building outlines)

[1187] Step 4: Based on the features extracted by the server, a 3D model is generated using the generation module.

[1188] The server uses 3D modeling software such as Blender or Unity to generate 3D models based on the extracted features, for example recreating the streets of Edo-period Tokyo based on the location information of detected rivers and roads.

[1189] Input: Extracted geographic and building features

[1190] Output: Generated 3D model

[1191] Step 5: The server sends the generated 3D model to the smart glasses

[1192] The server then transmits the generated 3D model to the smart glasses via wireless communication such as Wi-Fi or Bluetooth.

[1193] Input: Generated 3D model

[1194] Output: 3D model sent to smart glasses

[1195] Step 6: The smart glasses display the 3D model and the user experiences it.

[1196] The smart glasses then display the received 3D model in the user's field of vision, giving the user the experience of being transported back in time to Edo-period Tokyo.

[1197] Input: 3D model sent to smart glasses

[1198] Output: A 3D model displayed in the user's field of view.

[1199] In this way, the system allows users to experience past streetscapes in real time.

[1200] (Application example 1)

[1201] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1202] In modern brick-and-mortar stores, it is difficult for users to experience the streetscapes and buildings of the past. This limits opportunities to understand historical background and culture. In addition, there is a lack of ways to visually reproduce past scenery in tourist spots and historical places, making it difficult to attract users' interest.

[1203] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscape on the smart glasses, and a means for allowing the user to experience past streetscapes in a real store. This allows the user to visually experience the appearance of past streetscapes and buildings in a real store, and to gain a deeper understanding of the historical background and culture.

[1204] "Smart glasses" are glasses-type devices that users can wear to provide augmented reality and display information.

[1205] An "antique map" is a map that shows geographical information from the past, recording historical geographical features and the layout of buildings.

[1206] "Old photographs" are photographs of landscapes, buildings, and people from the past, providing visual historical information.

[1207] "Means of analysis" refers to technologies and devices for digitally processing information from old maps and old photographs and extracting geographical and architectural features.

[1208] "Means of generation" refers to technology and equipment for creating 3D models of past cityscapes and buildings based on analyzed information.

[1209] "Display means" refers to the technology or devices for displaying the generated 3D model on the smart glasses and providing the user with a visual experience.

[1210] A "brick and mortar store" is a store that exists in a physical location and where users can visit in person to purchase goods or services.

[1211] "Means for experiencing streetscapes of the past" refers to technologies and devices that allow users to visually experience streetscapes and buildings of the past through smart glasses.

[1212] The system for implementing the present invention includes smart glasses, a server, an analysis module, a generation module, and a display module. Detailed processing of each module and the hardware and software used will be described below.

[1213] 1. System Programming

[1214] The system includes a program that works in conjunction with smart glasses to analyze information from old maps and old photographs, generate a 3D model of the cityscape of the past, and display it to the user.

[1215] 2. Program Processing Explanation

[1216] Hardware used:

[1217] Smart glasses (e.g. Microsoft HoloLens, Google Glass)

[1218] Server (a computer with high-performance data processing capabilities)

[1219] Software used:

[1220] OpenCV: Image analysis library

[1221] ARModule: Augmented reality display module

[1222] OldMapAnalyzer: Old map analysis module

[1223] ThreeDModelGenerator: 3D model generation module

[1224] Process flow:

[1225] 1. Loading old maps:

[1226] The server uses OpenCV to read image data from old maps and old photographs, thereby obtaining past geographical information and architectural features as digital data.

[1227] 2. Feature Extraction:

[1228] The server uses the OldMapAnalyzer module to extract geographic and architectural features from old maps and photographs, providing detailed information about past streetscapes.

[1229] 3. 3D model generation:

[1230] The server uses the ThreeDModelGenerator module to generate a 3D model based on the extracted features, which allows the past streetscape and buildings to be reproduced in three dimensions.

[1231] 4.Display:

[1232] The server transmits the generated 3D model to the smart glasses via ARModule and displays it to the user, allowing the user to visually experience the streetscape of the past in a brick-and-mortar store.

[1233] 3. Specific Examples

[1234] For example, when a user visits a historical tourist site, they can wear the smart glasses and experience the townscape of the Edo period, allowing them to enjoy sightseeing while comparing modern scenery with past scenery.

[1235] Example prompt:

[1236] "Please recreate the streets of Asakusa in the Edo period. Analyze old maps and old photographs, generate a 3D model and display it on the smart glasses."

[1237] In this way, users can enjoy shopping and sightseeing while comparing modern-day Asakusa with Asakusa of the Edo period.

[1238] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1239] Step 1:

[1240] The server uses OpenCV to read image data of old maps and old photographs. It receives the image file path of the old map or old photograph as input and obtains image data as output. This image data is used for subsequent analysis processing.

[1241] Step 2:

[1242] The server uses the OldMapAnalyzer module to extract geographic and architectural features from the loaded image data. As input, it receives the image data acquired in step 1, and as output, it generates a dataset containing geographic and architectural features. This dataset will be the basis for reconstructing the streetscape of the past.

[1243] Step 3:

[1244] The server uses the ThreeDModelGenerator module to generate a 3D model based on the extracted features. It receives as input the dataset containing the geographical and architectural features generated in step 2, and generates as output 3D model data. This 3D model data is used to reproduce the appearance of past streets and buildings in three dimensions.

[1245] Step 4:

[1246] The server sends the generated 3D model data to the smart glasses through ARModule. It receives the 3D model data generated in step 3 as input and sends data to be displayed on the smart glasses as output. This data is used for the user to visually experience the past cityscape through the smart glasses.

[1247] Step 5:

[1248] The user wears the smart glasses and experiences the streetscape of the past in a real store. As input, the user receives 3D model data displayed on the smart glasses, and as output, the user visually experiences the streetscape and buildings of the past. Through this experience, the user can enjoy shopping and sightseeing while comparing the modern landscape with the landscape of the past.

[1249] Example 2

[1250] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1251] In order to provide an experience that combines modern streetscapes with past streetscapes, technology is required to acquire appropriate historical data based on the user's current location, and analyze and generate it in real time. However, with conventional technology, there is no system that consistently acquires location information, analyzes historical data, and displays the generated streetscapes, making it difficult for users to smoothly experience past streetscapes.

[1252] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1253] In this invention, the server includes a means for acquiring an old map related to the vicinity of the location information, which is a map from a time before the present, and an old photograph related to the vicinity of the location information, based on the user's location information, a means for extracting geographical features and architectural features from the acquired old map and old photograph, a prompt sentence for instructing to generate a 3D model representing a landscape related to the vicinity of the location information, which is a landscape from a time before the present, based on the extracted geographical features and architectural features, a means for generating the 3D model based on a generative AI model, and a means for displaying an image obtained based on the generated 3D model on smart glasses worn by the user. This allows the user to smoothly enjoy an experience in which modern cityscapes and past cityscapes are blended in real time.

[1254] "Smart glasses" are wearable devices that, when worn by a user, can overlay digital information onto images from the real world.

[1255] “Location Information” means data that describes a User’s current geographic location, such as data obtained using GPS or other location measurement technologies.

[1256] A "server" is a computer system that processes and stores data via a network, and receives, analyzes, and generates information sent from a user's device.

[1257] An "antique map" is a map that shows geographical information from the past, and visually represents historical cityscapes and topography.

[1258] "Old photographs" are photographs taken in times past that record historical buildings and landscapes.

[1259] "Analysis" is the process of examining acquired data in detail and extracting useful information from it.

[1260] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate new data and content, generating past streetscapes based on a prompt text.

[1261] A "prompt" is an instruction entered into a generative AI model to cause it to generate a specific output.

[1262] "Streetscape" refers to the layout and appearance of buildings, roads, and other structures in a particular area.

[1263] "Display" refers to the process of visually presenting digital information to a user by overlaying the information on the smart glasses display.

[1264] This invention is a system that allows a user to wear smart glasses and stand at a specific location, and generates an old streetscape based on information from old maps and old photographs of the location, providing an experience that blends the old streetscape with the real world. A specific embodiment of this system will be described below.

[1265] First, a user puts on the smart glasses and stands at a specific location, for example, in Asakusa, Tokyo. The smart glasses use the built-in GPS sensor to obtain the user's current location information. This location information is then sent to the server in real time.

[1266] The server receives the location information sent from the device. Based on the received location information, it uses geographic information services such as Google Maps API and OpenStreetMap API to obtain old maps and old photographs of the location. The obtained data is analyzed within the server, and geographical and architectural features are abstracted.

[1267] Based on the analyzed data, the server generates a prompt sentence to be input to a generative AI model (e.g., GPT-4 (registered trademark) or DALL-E). For example, the server creates a prompt sentence such as, "Based on geographical features and architectural features, please generate a 3D model that represents the scenery of the Edo period around the current location."

[1268] The generative AI model generates a 3D model representing the old scenery (cityscape) based on the prompt sent from the server. The generated data is returned to the server. The server receives the 3D model data representing the old scenery returned from the generative AI model and sends it to the terminal.

[1269] The device receives 3D model data representing old scenery sent from the server. The images obtained from the received data are superimposed on the real-world image on the smart glasses display, allowing users to enjoy an experience in which the real world and old townscapes are blended in real time.

[1270] As a concrete example, consider the case where a user stands in front of Kaminarimon Gate in Asakusa. When the user wears smart glasses and stands at a specific point in Asakusa, the smart glasses send the location information of that point to the server. The server uses the Google Maps API to obtain old maps and old photos of Asakusa, and inputs the following prompt text to the generative AI model:

[1271]

[1272] The generative AI model generates an image of Asakusa in the Edo period based on this prompt and returns the data to the server. The server then sends the generated data of Asakusa in the Edo period to the device. The device then overlays the received data with real-world images and displays them on the smart glasses display. The user can experience an image that combines modern Asakusa with Asakusa in the Edo period.

[1273] This system allows users to smoothly enjoy an experience in real time where modern cityscapes blend with historic streetscapes.

[1274] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1275] Step 1:

[1276] A user puts on the smart glasses and stands at a particular point.

[1277] Input: User's location

[1278] Output: Smart glasses get location information

[1279] Specific operation: A user wears the smart glasses and stands at a specific location, for example, in Asakusa, Tokyo. The smart glasses use the built-in GPS sensor to obtain the user's current location.

[1280] Step 2:

[1281] The device obtains location information and sends it to the server.

[1282] Input: Obtained location information

[1283] Output: Location sent to the server

[1284] How it works: The smart glasses transmit the acquired location information to the server in real time. The location information is sent to the server in the form of latitude and longitude.

[1285] Step 3:

[1286] The server receives the location information and retrieves the corresponding old maps and old photographs.

[1287] Input: Location sent

[1288] Output: Obtained old maps and old photographs

[1289] Specific operation: The server receives the location information sent from the device. Based on the received location information, it uses geographic information services such as Google Maps API and OpenStreetMap API to obtain old maps and old photos of that location.

[1290] Step 4:

[1291] The server analyzes the acquired data and inputs a prompt sentence along with the analysis results into the generative AI model.

[1292] Input: Obtained old maps and old photographs

[1293] Output: Prompt sentence to be input to the generative AI model

[1294] Specific operation: The server analyzes the acquired data of old maps and old photographs. Along with the analysis results, it generates a prompt sentence to be input to a generative AI model (e.g., GPT-4 or DALL-E). For example, it creates a prompt sentence such as, "Based on geographical features and architectural features, please generate a 3D model that represents the scenery of the Edo period around the current location."

[1295] Step 5:

[1296] The generative AI model generates a 3D model representing the old cityscape and returns the data to the server.

[1297] Input: prompt statement

[1298] Output: Generated 3D model data showing the old streetscape

[1299] Specific operation: The generative AI model generates old streetscapes based on the prompts sent from the server. The generated data is returned to the server.

[1300] Step 6:

[1301] The server transmits the generated data to the terminal.

[1302] Input: Generated old streetscape data

[1303] Output: Data sent to the terminal

[1304] Specific operation: The server receives the data of old cityscapes returned by the generative AI model and sends it to the device.

[1305] Step 7:

[1306] The data received by the device is overlaid on images of the real world and displayed.

[1307] Input: Received data of old streetscapes

[1308] Output: Fusion image displayed on the smart glasses display

[1309] Specific operation: The device receives data of old streetscapes sent from the server, and displays the received data on the smart glasses display, overlaying it with images of the real world.

[1310] Step 8:

[1311] Users will experience a fusion of the real world and old-fashioned streetscapes.

[1312] Input: Fusion image displayed on the smart glasses display

[1313] Output: The user's experience

[1314] Specific operation: Through the smart glasses, the user experiences images that combine modern cityscapes with those of the past. For example, the user can enjoy images of modern Asakusa overlapping with those of Asakusa in the Edo period.

[1315] In addition, the user's emotion may be recognized using an emotion engine described later, and the display content of the smart glasses may be adjusted based on the recognized emotion of the user. Specifically, the color tone of the scenery and the facial expressions of people included in the displayed image may be adjusted according to the user's emotion.

[1316] (Application example 2)

[1317] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1318] Systems that allow users to visually experience a fusion of modern cityscapes and past cityscapes are extremely useful in the fields of tourism and education. However, with current technology, it is difficult for users to obtain, analyze, and visually display historical information about a specific location in real time when they stand there. In addition, technology that uses generative AI models to generate prompt sentences and display past cityscapes based on those sentences is still in its infancy. This poses the challenge of making it difficult for users to experience past cityscapes in real time.

[1319] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1320] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for acquiring information on old maps and old photographs of a specific location when the user stands at the location, a means for generating a prompt sentence using a generative AI model based on the acquired information, and a means for visually displaying the old streetscape based on the generated prompt sentence. This enables the user to experience the past streetscape of a specific location in real time simply by standing at the location.

[1321] "Smart glasses" are wearable devices that allow users to overlay digital information onto the real world.

[1322] An "antique map" is a map that shows geographical information from the past, recording historical topography and the layout of buildings.

[1323] "Old photographs" are photographs of landscapes, buildings, and people from the past, and serve as visual records of historical information.

[1324] "Means of analysis" refers to techniques and methods for processing the information from acquired old maps and old photographs as data and extracting geographical and architectural features.

[1325] "Means of generation" refers to the techniques and methods for recreating past streetscapes and landscapes as digital data based on analyzed information.

[1326] "Display means" refers to the technology or method for displaying the generated past cityscapes and scenery on the smart glasses.

[1327] A "particular location" is a location where a user is standing and about which historical information relating to that location is retrieved.

[1328] A "generative AI model" is a model that uses artificial intelligence to analyze data and generate output for a specific purpose.

[1329] A "prompt sentence" is an instruction sentence output by a generative AI model, and is text generated based on specific information.

[1330] "Means for visual display" refers to techniques or methods for displaying the generated past streetscapes or scenery so that the user can visually experience them.

[1331] The system for implementing the present invention includes smart glasses, a server, and a generative AI model. A specific embodiment of the system is described below.

[1332] First, when a user wears the smart glasses and stands at a specific location, the smart glasses send the location information of that location to the server. Based on the location information, the server retrieves information on old maps and old photographs related to that location from a database that stores past geographical information and historical photographs.

[1333] Next, the server analyzes the information on the acquired old maps and old photographs. Image processing techniques are used for the analysis to extract geographical and architectural features. Specifically, image processing libraries such as OpenCV are used to extract feature points from maps and photographs.

[1334] Based on the analyzed information, the server generates a prompt sentence using a generative AI model. For the generative AI model, a natural language generation model such as GPT-3 is used. The generated prompt sentence is an instruction sentence for reproducing the past streetscape.

[1335] Based on the generated prompt, the server generates digital data of the streetscape of the past. This generation is done using 3D modeling software and rendering engines, such as Blender and Unity.

[1336] The generated past streetscapes are sent to the smart glasses and displayed in the user's field of view, allowing the user to have a visual experience that blends the real world with the past streetscapes.

[1337] As a concrete example, when a user stands at Tokyo Station, the server retrieves old photos of Tokyo Station, analyzes them, and generates a prompt. The generated prompt is "Please show me the old streetscape of Tokyo Station." Based on this prompt, the server generates an old streetscape of Tokyo Station and displays it on the smart glasses. The user can enjoy walking around while comparing the modern Tokyo Station with the old streetscape of Tokyo Station.

[1338] In this way, the present invention enables a user to experience a past streetscape of a particular location in real time, simply by standing at that location.

[1339] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1340] Step 1:

[1341] A user puts on the smart glasses and stands at a particular point.

[1342] Input: User's location

[1343] Output: Location information is sent from the smart glasses to the server.

[1344] Specific operation: The GPS function of the smart glasses obtains the user's current location and sends that location information to the server.

[1345] Step 2:

[1346] Based on the location information, the server retrieves information on old maps and old photographs related to that location from a database.

[1347] Input:Location

[1348] Output: Information on old maps and old photographs

[1349] Specific operation: The server queries the database using the location information as a key and retrieves the relevant data on old maps and old photographs.

[1350] Step 3:

[1351] The server analyzes the information on old maps and old photographs it has acquired.

[1352] Input: Information on old maps and old photographs

[1353] Output: Geographical and architectural features

[1354] Specific operation: The server uses image processing libraries such as OpenCV to extract feature points from old maps and old photographs.

[1355] Step 4:

[1356] The server generates a prompt sentence based on the parsed information using a generative AI model.

[1357] Input: Geographical and architectural features

[1358] Output: Prompt statement

[1359] Specific operation: The server inputs feature data into a generative AI model such as GPT-3 and generates prompt sentences to recreate past cityscapes.

[1360] Step 5:

[1361] The server generates past streetscapes as digital data based on the generated prompt sentence.

[1362] Input: Prompt statement

[1363] Output: Digital data of past streetscapes

[1364] Specific operation: The server uses 3D modeling software such as Blender or Unity to generate a 3D model based on the prompt text.

[1365] Step 6:

[1366] The server transmits the generated past streetscapes to the smart glasses and displays them in the user's field of view.

[1367] Input: Digital data of past streetscapes

[1368] Output: A past cityscape displayed on smart glasses

[1369] Specific operation: The server sends the generated 3D model to the smart glasses, which overlay it on the user's field of view.

[1370] In this way, users can have a visual experience that blends the real world with past cityscapes.

[1371] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1372] "Example 1"

[1373] The present invention works in conjunction with smart glasses to analyze information from old maps and old photographs, and uses the analyzed information to

[1374] The system generates old streetscapes based on this data and displays the generated streetscapes on the smart glasses. It also includes an emotion engine that recognizes the user's emotions. This emotion engine infers the user's emotions from the user's facial expressions, tone of voice, and operation patterns of the smart glasses. Based on the inferred emotions, the system adjusts the color tones and music of the generated streetscapes, the facial expressions of people displayed, and other aspects to provide an experience that matches the user's emotions.

[1375] "Example 2"

[1376] As a specific example, when a user wears smart glasses and stands at a specific location, information on old maps and old photographs of that location is sent to the system. An analysis module in the system extracts geographical and architectural features from the sent information and generates old streetscapes based on the extracted features. The generated streetscape is displayed on the smart glasses, and the user experiences a fusion of the real world and old streetscapes. At this time, the emotion engine recognizes the user's emotions, such as joy, surprise, and nostalgia, and adjusts the display of the streetscape according to those emotions. For example, when the user is feeling joy, the colors of the streetscape are made vivid and the people's expressions are brightened. Conversely, when the user is feeling nostalgic, the colors of the streetscape are made subdued and the people's expressions are made gentler.

[1377] The process flow of each embodiment will be described below.

[1378] "Example 1"

[1379] Step 1: A user puts on the smart glasses and stands at a specific point.

[1380] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[1381] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[1382] Step 4: Generate old streetscapes based on the extracted features.

[1383] Step 5: Display the generated cityscape on the smart glasses.

[1384] Step 6: The emotion engine recognizes the user's emotion and adjusts the display of the cityscape according to that emotion.

[1385] "Example 2"

[1386] Step 1: A user puts on the smart glasses and stands at a specific point.

[1387] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[1388] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[1389] Step 4: Generate old streetscapes based on the extracted features.

[1390] Step 5: Display the generated cityscape on the smart glasses.

[1391] Step 6: The emotion engine recognizes the user's emotion and adjusts the display of the streetscape according to that emotion. For example, when the user is feeling happy, the colors of the streetscape are made vivid and the people's expressions brighter. On the other hand, when the user is feeling nostalgic, the colors of the streetscape are made muted and the people's expressions calmer.

[1392] Example 1

[1393] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1394] With conventional technology, it was difficult to analyze information from old maps and old photographs and recreate old streetscapes. In addition, it was not possible to adjust the display content according to the user's emotions, making it impossible to provide a consistent experience to the user. Furthermore, there was a lack of technology to change the display content in real time in conjunction with smart glasses. This led to problems such as a decrease in the sense of immersion and satisfaction when users experienced historical streetscapes.

[1395] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1396] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for recognizing a user's emotion, and a means for adjusting the display content based on the recognized emotion. This allows a user to experience old streetscapes in real time based on information on old maps and old photographs, and further allows the experience to be adjusted according to the user's emotion.

[1397] "Smart glasses" are wearable devices that can display visual information as augmented reality when worn by a user.

[1398] An "antique map" is a map that shows geographical information from the past, including historical geographical features and the layout of buildings.

[1399] "Old photographs" are photographs taken in the past that record historical buildings and landscapes.

[1400] An "analysis module" is software or hardware used to analyze information from old maps and old photographs and extract geographical and architectural features.

[1401] A "3D model" is a three-dimensional digital model that is generated based on analyzed geographical features and architectural characteristics.

[1402] An "emotion engine" is software or hardware that recognizes and estimates a user's emotions from the user's facial expressions, tone of voice, operation patterns, etc.

[1403] The "means for adjusting the display content" refers to software or hardware for changing the color tones of the displayed streetscape, the music, the facial expressions of people displayed, etc., based on the recognized emotions of the user.

[1404] "Geographic features" refers to geographic elements such as rivers, roads, and mountains on a map or photograph.

[1405] "Building features" refers to elements such as the shape, layout, and structure of a building on a map or photograph.

[1406] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1407] This invention is a system that works in conjunction with smart glasses, analyzes information from old maps and old photographs, generates old streetscapes based on the analyzed information, and displays the generated streetscapes on the smart glasses. It also includes a function to recognize the user's emotions and adjust the display content based on the emotions.

[1408] Hardware and Software Used

[1409] The server uses the following hardware and software:

[1410] Image analysis library: OpenCV, TensorFlow

[1411] 3D modeling software: Unity, Unreal Engine

[1412] Emotion Recognition API: Microsoft Azure Emotion API, Google Cloud Speech-to-Text API

[1413] Communication technology: Wi-Fi, Bluetooth

[1414] Data processing and calculation

[1415] The server receives data of old maps and old photographs uploaded by users. The received data is analyzed using image analysis libraries (OpenCV and TensorFlow). The analysis module extracts geographical features (rivers, roads, mountains, etc.) and architectural features (shape, layout, structure, etc. of buildings).

[1416] Based on the analysis results, the server uses 3D modeling software (Unity or Unreal Engine) to generate a 3D model. The generated 3D model reproduces the old streetscape. For example, when recreating a streetscape from the Edo period, the layout of the buildings and roads at that time is faithfully reproduced.

[1417] The generated 3D model is then sent to the smart glasses via Wi-Fi or Bluetooth, where the smart glasses display the received 3D model, giving the user the experience of traveling back in time.

[1418] Emotion recognition and display content adjustment

[1419] The server recognizes the user's emotions using an emotion engine. The emotion engine estimates the user's emotions from the user's facial expressions, tone of voice, and operation patterns of the smart glasses. For example, it uses Microsoft Azure's Emotion API or Google Cloud's Speech-to-Text API.

[1420] After the emotion engine estimates the user's emotion, the server adjusts the display content based on that emotion. For example, if the server estimates that the user is having fun, it will brighten the colors of the streetscape and play cheerful music. It will also adjust the facial expressions of people displayed to smile.

[1421] Examples of concrete examples and prompts

[1422] For example, if a user uploads old maps and old photographs from the Edo period, the server receives this data and uses an analysis module to extract geographical and architectural features from the Edo period. Based on the analysis results, a 3D model of the Edo period streetscape is generated using Unity. The generated 3D model is sent to the smart glasses via Wi-Fi, and the user can experience the Edo period streetscape through the smart glasses.

[1423] When a user is having fun walking through a street, the emotion engine infers from the user's facial expression that the user is having fun. The server brightens the colors of the street and plays cheerful music. The server also adjusts the facial expressions of the people displayed to smile.

[1424] Example of a prompt to be input to a generative AI model:

[1425] "Analyze old maps and photographs from the Edo period and generate a 3D model of the streets from that period. When the user is enjoying it, brighten the colors of the streets and play fun music."

[1426] In this way, a system is realized in which the server, terminal, and user work together to provide the user with an experience of historical townscapes that correspond to their emotions.

[1427] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1428] Step 1:

[1429] Users upload old maps and old photographs.

[1430] A user uploads data of old maps and old photographs using a terminal (e.g., a smartphone or tablet) connected to the smart glasses. As input, they provide image files in JPEG or PNG format. As output, the uploaded data is sent to the server.

[1431] Step 2:

[1432] The server receives the uploaded data.

[1433] The server receives data of old maps and old photographs uploaded by users. As input, it receives image files sent by users. As output, it stores the received data in a database in the server.

[1434] Step 3:

[1435] The server analyzes the data using an analysis module.

[1436] The server starts the analysis module and analyzes the received data of old maps and old photographs. It uses the stored image files as input. The analysis module uses image analysis libraries such as OpenCV and TensorFlow to extract geographical and architectural features. The extracted feature information is obtained as output.

[1437] Step 4:

[1438] The server generates a 3D model based on the analysis results.

[1439] The server uses 3D modeling software such as Unity or Unreal Engine to generate a 3D model based on the feature information obtained from the analysis module. It uses the extracted feature information as input. As output, a 3D model that reproduces the old streetscape is generated.

[1440] Step 5:

[1441] The server transmits the generated 3D model to the smart glasses.

[1442] The server sends the generated 3D model to the smart glasses. As input, the server uses the generated 3D model. As output, the 3D model is sent to the smart glasses. For transmission, wireless communication technologies such as Wi-Fi and Bluetooth are used.

[1443] Step 6:

[1444] Smart glasses display 3D models.

[1445] The smart glasses display the 3D model received from the server. As input, they use the 3D model sent from the server. As output, the user can visually experience the old streetscape as if they had traveled back in time.

[1446] Step 7:

[1447] Users experience the cityscape through smart glasses.

[1448] A user can wear the smart glasses and walk around the displayed 3D model of the cityscape. As input, we use the 3D model displayed on the smart glasses. As output, the user experiences the historical cityscape.

[1449] Step 8:

[1450] The server uses an emotion engine to recognize the user's emotion.

[1451] The server starts an emotion engine and recognizes the user's emotions from the user's facial expressions, tone of voice, and operation patterns of the smart glasses. The inputs used are the user's facial expression data, voice data, and operation data. The emotion engine uses Microsoft Azure's Emotion API and Google Cloud's Speech-to-Text API. The output is an estimated user's emotions.

[1452] Step 9:

[1453] The server adjusts the display content based on the user's emotions.

[1454] The server adjusts the display content of the generated 3D model based on the user's emotions estimated by the emotion engine. As input, it uses the estimated user's emotion data. As output, the color tones of the streetscape, music, and facial expressions of people displayed are adjusted. For example, if it is estimated that the user is having fun, the color tones of the streetscape are brightened and cheerful music is played. The facial expressions of people displayed are also adjusted to smile.

[1455] (Application example 1)

[1456] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1457] Conventional smart glasses-based systems can analyze old maps and old photographs to recreate old streetscapes, but they cannot adjust the display content according to the user's emotions, so the user experience is uniform and it is difficult to provide an optimized experience for each individual user. In addition, there is a lack of concrete measures to improve the shopping experience in physical stores. This makes it difficult to increase user satisfaction and stimulate purchasing desire in physical stores.

[1458] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1459] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for recognizing the user's emotions, and a means for adjusting the display content based on the recognized emotions. This makes it possible to provide optimal display content according to the user's emotions, improving the shopping experience in a brick-and-mortar store and increasing the user's satisfaction and purchasing motivation.

[1460] "Smart glasses" are wearable devices that can display visual information as augmented reality when worn by the user.

[1461] An "antique map" is a map that shows geographical information from the past, recording the locations of historical geographical features and buildings.

[1462] "Old photographs" are photographs taken in the past that record historical buildings and landscapes.

[1463] "Means of analysis" refer to the technical means for analyzing information from old maps and old photographs as digital data and extracting geographical and architectural features.

[1464] The "means of generation" refers to the technical means for generating a three-dimensional model of a past streetscape based on the analyzed information.

[1465] "Means for displaying" means the technical means for displaying the generated 3D model on the smart glasses and providing visual information to the user.

[1466] "Means of recognizing emotions" refers to technical means for inferring emotions from a user's facial expressions, tone of voice, operation patterns, etc.

[1467] "Means for adjusting the display content" refers to technical means for adjusting the color tones of the displayed cityscape, the music, the facial expressions of people displayed, etc., based on the recognized emotions of the user.

[1468] The system for implementing this invention uses hardware such as smart glasses, a server, a camera, a microphone, etc., and software such as OpenCV, TensorFlow, Affectiva SDK, Unity, FMOD, etc. The specific configuration and operation of the system are described below.

[1469] System Configuration

[1470] 1. Smart glasses:

[1471] It is a wearable device that is worn by the user and displays visual information as augmented reality.

[1472] 2. Server:

[1473] It is a central processing device that analyzes information from old maps and old photographs and generates old cityscapes based on the analyzed information.

[1474] 3. Camera:

[1475] A device for capturing a user's facial expressions.

[1476] 4. Microphone:

[1477] A device for capturing the tone of a user's voice.

[1478] Software configuration

[1479] 1. OpenCV:

[1480] This is a library for analyzing images of old maps and old photographs.

[1481] 2. TensorFlow:

[1482] A framework for running machine learning models for image analysis.

[1483] 3. Affectiva SDK:

[1484] It is a software development kit for recognizing user emotions.

[1485] 4. Unity:

[1486] It is a game engine for generating 3D models and adjusting the display content.

[1487] 5. FMOD:

[1488] An audio engine for controlling music and sound effects.

[1489] System Operation

[1490] 1. Analysis of old maps and photographs:

[1491] The server uses OpenCV and TensorFlow to analyze old maps and photographs and extract geographical and architectural features.

[1492] 2. Generate 3D model:

[1493] The server uses Unity to generate a 3D model of the old townscape based on the extracted features.

[1494] 3. Emotion Recognition:

[1495] The server analyzes the data obtained from the camera and microphone using the Affectiva SDK to recognize the user's emotions.

[1496] 4. Adjust the display:

[1497] The server uses Unity and FMOD to adjust the content displayed based on the emotions it recognizes, such as changing the color tones of the cityscape, the music, and the facial expressions of people displayed.

[1498] Examples

[1499] If a user wants to experience the streets of the Edo period, they can wear the smart glasses and walk around a brick-and-mortar store. The streets of the Edo period, recreated from old maps and photographs, are displayed on the smart glasses, giving the user the experience of traveling back in time to that era. Furthermore, if the user has a happy expression, the colors of the streets become brighter and cheerful music plays.

[1500] Examples of prompt statements

[1501] "Develop an application for smart glasses that recreates the streets of the Edo period and adjusts the display content according to the user's emotions. Analyze old maps and old photographs, generate 3D models, and use an emotion recognition engine to adjust the display content."

[1502] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1503] Step 1:

[1504] The server receives image data of old maps and old photographs. As input, image files of old maps and old photographs are provided. The server reads these images using OpenCV and performs preprocessing. Specifically, it removes noise from the images and adjusts the resolution. As output, it obtains the preprocessed image data.

[1505] Step 2:

[1506] The server inputs the preprocessed image data into a TensorFlow model to extract geographic and architectural features. The preprocessed image data is provided as input. The server uses TensorFlow to perform image analysis and extract features. Specifically, it identifies building outlines and topographical features. The extracted feature data is obtained as output.

[1507] Step 3:

[1508] The server uses Unity to generate a 3D model based on the extracted feature data. The feature data is provided as input. The server executes Unity scripts to generate a 3D model that reproduces the past streetscape. Specifically, it reproduces the layout of buildings and the terrain. The generated 3D model data is obtained as output.

[1509] Step 4:

[1510] The user wears the smart glasses and transmits real-time data to the server through the camera and microphone. As input, the user's facial expression data and voice data are provided. The server uses the Affectiva SDK to analyze these data and recognize the user's emotions. Specifically, it analyzes changes in facial expressions and tone of voice. As output, it obtains the recognized emotion data.

[1511] Step 5:

[1512] The server uses Unity and FMOD to adjust the display content based on the recognized emotion data. Emotion data and 3D model data are provided as input. The server executes Unity scripts to adjust the color tone of the cityscape, the music, and the facial expressions of the people displayed. Specifically, it brightens the color tone and changes the music depending on the emotion. The adjusted display content data is obtained as output.

[1513] Step 6:

[1514] The server transmits the adjusted display content data to the smart glasses to display it to the user. As an input, the adjusted display content data is provided. The server transmits data to the display of the smart glasses to provide visual information to the user. Specifically, the reproduced cityscape and the adjusted display content are displayed on the smart glasses. As an output, visual information experienced by the user is obtained.

[1515] Example 2

[1516] Next, a second embodiment of the second form will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[1517] In conventional systems, it was difficult for a user to visually reproduce the past streetscape of a particular location when the user stood at that location. In addition, the display content could not be dynamically adjusted according to the user's emotions, limiting the user experience. This led to the issue that users could not experience the past streetscape more realistically.

[1518] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1519] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, and a means for recognizing the user's emotions and adjusting the display according to the emotions. This makes it possible to visually reproduce the past streetscape of a specific location when the user stands at that location in real time, and further dynamically adjust the display content according to the user's emotions.

[1520] "Smart glasses" are wearable devices that, when worn by a user, can overlay digital information onto visual information from the real world.

[1521] An "antique map" is a map that shows geographical information from the past, and includes historical geographical information for specific locations or areas.

[1522] An "archive photograph" is a photograph taken in the past that contains historical visual information of a particular location or area.

[1523] "Means of analysis" refers to the technical means for receiving information from old maps and old photographs and extracting geographical and architectural features from that information.

[1524] The "means of generation" refers to the technical means for recreating the streetscape of the past based on the analyzed information, specifically the generation of a three-dimensional model.

[1525] The "display means" refers to the technical means for displaying the generated past streetscape on the smart glasses.

[1526] "Means for recognizing emotions" refers to a technical means for detecting emotions from a user's facial expressions, voice, etc., and adjusting the display content based on those emotions.

[1527] This invention is a system in which a user wears smart glasses and stands at a specific location, which visually recreates the past streetscape of that location, and dynamically adjusts the display content according to the user's emotions.

[1528] The server receives information on old maps and old photos sent from the smart glasses when a user stands at a specific location. The received information is passed to the analysis module, which extracts geographical and architectural features. The analysis module generates old streetscapes based on the extracted features. A generative AI model is used for this generation. The generated streetscape data is then sent to the smart glasses.

[1529] When a user stands at a specific spot, the device (smart glasses) sends information about old maps and old photographs of that spot to the server. The device receives data on old streetscapes generated by the server and displays it blended with the real world. The device also uses an emotion engine to recognize the user's emotions and adjusts the display according to those emotions. For example, if the user is feeling happy, the colors of the streetscape will be made more vivid and the people's expressions will be brighter. Conversely, if the user is feeling nostalgic, the colors of the streetscape will be made more subdued and the people's expressions will be made calmer.

[1530] The user puts on the smart glasses and stands at a specific location. The glasses send information about old maps and old photos of that location to the server, and receive the old streetscape generated by the server. The user can enjoy an experience where the real world and the old streetscape are fused together. In addition, the user's emotions are recognized and the display is adjusted according to those emotions, providing a more moving experience.

[1531] As a concrete example, let's say a user is standing in Asakusa, Tokyo. When the user wears smart glasses and stands in front of Kaminarimon Gate in Asakusa, the smart glasses send information about old maps and old photographs of that location to the server. The server analyzes the received information and generates the streetscape of Asakusa in the Edo period. The generated streetscape data is sent to the smart glasses as a 3D model. The smart glasses display the Edo period streetscape superimposed on the real scenery in front of Kaminarimon Gate. If the user is feeling happy, the colors of the Edo period streetscape will become more vivid and people's expressions will become brighter. On the other hand, if the user is feeling nostalgic, the colors of the streetscape will become more subdued and people's expressions will become calmer.

[1532] Examples of prompts to be input to a generative AI model include the following:

[1533] "When a user stands in Asakusa, Tokyo, generate an image of the streets of Asakusa from the Edo period and display it on the smart glasses. Adjust the display according to the user's emotions. If the user is feeling joy, make the colors vivid and brighten the people's expressions. If the user is feeling nostalgia, make the colors calmer and calm the people's expressions."

[1534] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1535] Step 1:

[1536] A user puts on the smart glasses and stands at a specific point.

[1537] Specific operation: The user wears the smart glasses and stands in front of the Kaminarimon Gate in Asakusa, Tokyo, for example. The smart glasses acquire the user's location information.

[1538] Input: User's location

[1539] Output: Get location information

[1540] Step 2:

[1541] The terminal (smart glasses) sends information about old maps and old photographs of that location to the server.

[1542] How it works: The smart glasses use GPS and Wi-Fi location information to determine the user's current location and send that information to a server.

[1543] Input: User's location

[1544] Output: Send location information

[1545] Step 3:

[1546] The server passes the received information to an analysis module, which extracts geographical and architectural features.

[1547] Specific operation: The server receives information on old maps and old photographs sent from the terminal and passes it to the analysis module. The analysis module extracts geographical features (e.g., road layout and river location) and architectural features (e.g., building shape and height).

[1548] Input: Information on old maps and old photographs

[1549] Output: Extraction of geographic and architectural features

[1550] Step 4:

[1551] The server generates old streetscapes based on the extracted features.

[1552] Specific operation: Based on the features extracted by the analysis module, the server uses a generative AI model to generate old streetscapes. For example, it recreates the streetscape of Asakusa in the Edo period.

[1553] Input: Geographical and architectural features

[1554] Output: Generation of old streetscape

[1555] Step 5:

[1556] The server transmits the generated streetscape data to the terminal.

[1557] Specific operation: The server sends the generated data of the old cityscape to the terminal as a 3D model or image data.

[1558] Input: Data of old streetscape

[1559] Output: Sending streetscape data

[1560] Step 6:

[1561] The streetscape data received by the device is blended with the real world and displayed.

[1562] Specific operation: The device overlays the received streetscape data onto the real-world scenery, and displays the fusion of real-world scenery and the old streetscape on the smart glasses display.

[1563] Input: Streetscape data

[1564] Output: Real-world blended display

[1565] Step 7:

[1566] The device recognizes the user's emotions using an emotion engine and adjusts the display accordingly.

[1567] Specific operation: The device uses an emotion engine to recognize emotions from the user's facial expressions and voice. For example, if the user is feeling happy, the colors of the streetscape will be brighter and the people's expressions will be brighter. On the other hand, if the user is feeling nostalgic, the colors of the streetscape will be muted and the people's expressions will be calmer.

[1568] Input: User emotion data

[1569] Output: Adjusting display according to emotions

[1570] (Application example 2)

[1571] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[1572] In modern brick-and-mortar stores, there is a lack of ways to allow customers to visually experience past history and culture. In addition, technology to dynamically adjust display content according to customer emotions is also underdeveloped. This makes it difficult to increase customer interest and satisfaction.

[1573] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1574] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for recognizing a user's emotion, and a means for adjusting the display content according to the recognized emotion. This allows customers to visually experience past history and culture in a brick-and-mortar store, and the experience can be dynamically adjusted according to the customer's emotion.

[1575] "Smart glasses" are glasses-type devices that users can wear to experience augmented reality (AR) and virtual reality (VR).

[1576] An "antique map" is a map that shows geographical information from the past, and records historical cityscapes and topography.

[1577] "Old photographs" are photographs taken in the past that record historical buildings and landscapes.

[1578] "Means of analysis" refers to the technology and equipment required to process information from old maps and old photographs as digital data and to understand and analyze their contents.

[1579] "Means of generation" refers to technology and devices for digitally recreating past cityscapes and landscapes based on analyzed information.

[1580] "Display means" refers to the technology and devices for projecting the generated past cityscapes and scenery onto the display of smart glasses.

[1581] "Means of recognizing emotions" refers to technology or devices for detecting and analyzing emotions from a user's facial expressions, voice, etc.

[1582] "Means for adjusting the display content" refers to technology or devices for dynamically changing the color tone, brightness, etc. of the displayed past cityscapes or scenery in response to the recognized emotions of the user.

[1583] A system for implementing the present invention includes smart glasses, a server, and various software modules. Specific embodiments are described below.

[1584] System Configuration

[1585] The system includes smart glasses, a server, and the following software modules:

[1586] Image processing library (e.g. OpenCV)

[1587] Emotion recognition library (e.g. EmotionRecognizer)

[1588] Augmented reality display library (e.g. ARDisplay)

[1589] Custom modules for obtaining old maps and old photographs (e.g. historical_data)

[1590] Hardware

[1591] Smart glasses (e.g. Microsoft HoloLens)

[1592] Smart glasses with built-in camera

[1593] software

[1594] OpenCV: Image processing library

[1595] EmotionRecognizer: Emotion recognition library

[1596] ARDisplay: Augmented reality display library

[1597] historical_data: A custom module for retrieving historical maps and photographs.

[1598] Processing flow

[1599] 1. Acquire video from the smart glasses camera

[1600] The camera built into the smart glasses captures footage in real time.

[1601] 2. Get current location information

[1602] The GPS function of the smart glasses is used to obtain the user's current location.

[1603] 3. Obtain information on old maps and old photographs

[1604] Based on the acquired location information, the server retrieves corresponding data on old maps and old photographs from the historical_data module.

[1605] 4. Emotion recognition

[1606] The server uses the video captured by the smart glasses' camera to recognize the user's emotions using the EmotionRecognizer library.

[1607] 5. Adjusting the display content

[1608] Depending on the recognized emotion, the ARDisplay library is used to adjust the color tone and brightness of the displayed content (old maps and old photographs).

[1609] 6. AR display

[1610] Adjusted old maps and old photographs are overlaid on the display of the smart glasses.

[1611] Examples

[1612] When a user enters a traditional Japanese sweets shop, the smart glasses will show the shop's original appearance and manufacturing process from the past. If the user is feeling happy, the images displayed will be bright and vivid. If the user is feeling nostalgic, the images will be muted.

[1613] Examples of prompt statements

[1614] "When a user puts on the smart glasses and enters a long-established Japanese confectionery shop, please develop an application that displays the shop's history and the manufacturing process from the past. Please also add a function to adjust the display content according to the user's emotions."

[1615] The above is a specific embodiment for carrying out the present invention.

[1616] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1617] Step 1:

[1618] The server acquires video in real time from the smart glasses' camera.

[1619] Input: Video data from the camera built into the smart glasses

[1620] Output: Real-time video frames

[1621] Specific operation: The camera in the smart glasses captures the user's field of vision and sends the video data to the server.

[1622] Step 2:

[1623] The server obtains the user's current location information using the smart glasses' GPS function.

[1624] Input: GPS data from smart glasses

[1625] Output: Current location information (latitude and longitude)

[1626] Specific operation: The GPS sensor in the smart glasses obtains the current location information and sends the data to the server.

[1627] Step 3:

[1628] Based on the acquired location information, the server retrieves corresponding data on old maps and old photographs from the historical_data module.

[1629] Input: Current location

[1630] Output: Data of old maps and old photographs

[1631] Specific operation: The server sends a query to the historical_data module based on the location information and retrieves the relevant data on old maps and old photographs.

[1632] Step 4:

[1633] The server uses the video captured by the smart glasses' camera to recognize the user's emotions using the EmotionRecognizer library.

[1634] Input: Real-time video frames

[1635] Output: User emotion data (e.g., happiness, nostalgia)

[1636] Specific operation: The server inputs video frames into the EmotionRecognizer library and analyzes emotions from the user's facial expressions and voice.

[1637] Step 5:

[1638] The server uses the ARDisplay library to adjust the color and brightness of the displayed content (old maps and old photographs) according to the recognized emotion.

[1639] Input: User emotion data, data on old maps and old photographs

[1640] Output: Adjusted data of historical maps and photographs

[1641] Specific operation: Based on the emotion data, the server dynamically changes the color tone and brightness of old maps and old photographs using the ARDisplay library.

[1642] Step 6:

[1643] The server then overlays the adjusted old maps and old photographs onto the smart glasses' display.

[1644] Input: Adjusted historical map and photo data, real-time video frames

[1645] Output: Augmented reality video displayed on smart glasses

[1646] Specific operation: The server overlays the adjusted old maps and old photos onto real-time video and sends them to the smart glasses display for display.

[1647] The above are the specific process steps for carrying out the present invention.

[1648] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1649] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1650] Another example of generative AI is Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1651] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1652] [Fourth embodiment]

[1653] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1654] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1655] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[1656] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1657] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[1658] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[1659] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[1660] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1661] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1662] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1663] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1664] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1665] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1666] "Example 1"

[1667] As one embodiment of the present invention, a system that works with smart glasses is provided. The system includes a module that analyzes information on old maps and old photographs. The analysis module extracts geographical features and architectural features from the old maps and old photographs. A module that generates old streetscapes operates based on the extracted features. The generation module generates a 3D model based on the extracted geographical features and architectural features. The generated streetscape is displayed on the smart glasses, allowing the user to experience a time-traveling experience.

[1668] "Example 2"

[1669] For example, when a user wears smart glasses and stands at a specific location, information about old maps and old photographs of that location is sent to the system. The system analyzes the information and generates an old streetscape of that location. The generated streetscape is displayed on the smart glasses, allowing the user to experience a fusion of the real world and the old streetscape.

[1670] The process flow of each embodiment will be described below.

[1671] "Example 1"

[1672] Step 1: A user puts on the smart glasses and stands at a specific point.

[1673] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[1674] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[1675] Step 4: The module that generates old streetscapes runs based on the extracted features.

[1676] Step 5: The generated streetscape is displayed on the smart glasses, giving the user an experience that blends the real world with the old streetscape.

[1677] "Example 2"

[1678] Step 1: A user puts on the smart glasses and stands at a specific point.

[1679] Step 2: Information about the location, such as old maps and old photographs, is sent to the system.

[1680] Step 3: The analysis module in the system extracts geographical and architectural features from the transmitted information.

[1681] Step 4: The module that generates old streetscapes runs based on the extracted features.

[1682] Step 5: The generated streetscape is displayed on the smart glasses, giving the user an experience that blends the real world with the old streetscape.

[1683] Example 1

[1684] Next, a description will be given of Example 1 of the embodiment 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1685] With conventional technology, it was difficult to reproduce past streetscapes using old maps and old photographs. In addition, the means by which users could experience past streetscapes were limited. This restricted the use of the technology in fields such as the reproduction of historical streetscapes, education, and tourism.

[1686] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1687] In this invention, the server includes a means for a user to upload old maps and old photographs from a terminal, a means for the server to receive the uploaded data, a means for the server to analyze the data using an analysis module and extract geographical features and architectural features, a means for the server to generate a 3D model using a generation module based on the features extracted by the server, a means for the server to transmit the generated 3D model to the smart glasses, and a means for the smart glasses to display the 3D model and for the user to experience it. This allows the user to experience past streetscapes in real time.

[1688] A "user" is an individual or group that uses the system to upload old maps and old photographs and experience streetscapes of the past.

[1689] A "terminal" is a device that a user uses to upload old maps and old photographs, and includes smartphones, personal computers, etc.

[1690] "Server" refers to a computer system that receives data uploaded by users and performs analysis and generation of 3D models.

[1691] The "analysis module" is a software component that analyzes data from old maps and old photographs and extracts geographical and architectural features.

[1692] "Geographical features" are geographical elements such as rivers, roads, and mountains that are included in old maps and old photographs.

[1693] "Building characteristics" refers to information such as the shape and layout of buildings contained in old maps and photographs.

[1694] The "generative module" is a software component for generating 3D models based on extracted geographical and architectural features.

[1695] A "3D model" is a three-dimensional digital representation generated based on extracted features.

[1696] "Smart glasses" are a wearable device that allows users to visually experience three-dimensional models.

[1697] "Uploading" refers to the act of a user using a terminal to send data of old maps or old photographs to a server.

[1698] "Receiving" is the act of the server taking in data sent by the user.

[1699] "Transmitting" refers to the act of the server sending the generated 3D model to the smart glasses.

[1700] "Display" refers to the act of the smart glasses projecting the received 3D model into the user's field of vision.

[1701] An "experience" is an act in which a user can visually sense a streetscape of the past through the smart glasses.

[1702] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[1703] This invention is a system that allows users to experience past streetscapes using old maps and old photographs. This system includes a series of processes in which the user uploads old maps and old photographs from a terminal, the server analyzes the data, generates a 3D model, and displays it to the user through smart glasses.

[1704] Hardware and Software Used

[1705] Terminal

[1706] Users upload old maps and old photographs using devices such as smartphones and computers, allowing them to easily access the system and provide data.

[1707] server

[1708] The server receives data uploaded by users, analyzes it, and generates 3D models. The following software is installed on the server:

[1709] Analysis module: Uses Python's OpenCV library and machine learning frameworks such as TensorFlow to analyze data from old maps and photographs and extract geographic and architectural features.

[1710] Generation module: Using 3D modeling software such as Blender or Unity, a 3D model is generated based on the extracted features.

[1711] Smart Glasses

[1712] The smart glasses display the 3D model sent from the server in the user's field of vision, allowing the user to experience past streetscapes in real time.

[1713] Examples

[1714] For example, if a user uploads an old map of Tokyo in the Edo period in JPEG format, the server receives this data and uses OpenCV to detect the outlines of rivers, roads, and buildings.The server then uses Blender to generate a 3D model of the streets of Tokyo in the Edo period based on the detected features.This 3D model is sent to the smart glasses via Wi-Fi, allowing the user to experience Tokyo in the Edo period through the smart glasses.

[1715] Examples of prompt statements

[1716] Examples of prompts to be input to a generative AI model include the following:

[1717] "Analyze old maps of Tokyo from the Edo period and generate a 3D model of the cityscape from that time."

[1718] Using this prompt, the generative AI model can perform the necessary analysis and generation processes to provide the user with an experience that recreates a historical cityscape.

[1719] In this way, the present invention provides a system that allows a user to experience past streetscapes in real time.

[1720] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1721] Program processing flow

[1722] Step 1: User uploads old maps and old photos from their device

[1723] Users upload data on old maps and old photographs to the system using devices such as smartphones or computers. For example, an old map of Tokyo from the Edo period can be uploaded in JPEG format.

[1724] Input: Data of old maps and old photographs (e.g. JPEG format images)

[1725] Output: Uploaded data is sent to the server

[1726] Step 2: The server receives the uploaded data

[1727] The server receives data of old maps and old photographs uploaded by users. This data is stored in a specific directory on the server.

[1728] Input: Data on old maps and old photographs sent by users

[1729] Output: Data stored in a specific directory on the server

[1730] Step 3: The server analyzes the data using the analysis module to extract geographic and architectural features.

[1731] The server uses machine learning frameworks such as Python's OpenCV library and TensorFlow to analyze the uploaded data, for example to detect the outlines of rivers, roads, and buildings in old maps and extract their features.

[1732] Input: Data of old maps and old photographs stored on the server

[1733] Output: Extracted geographic and architectural features (e.g. rivers, roads, building outlines)

[1734] Step 4: Based on the features extracted by the server, a 3D model is generated using the generation module.

[1735] The server uses 3D modeling software such as Blender or Unity to generate 3D models based on the extracted features, for example recreating the streets of Edo-period Tokyo based on the location information of detected rivers and roads.

[1736] Input: Extracted geographic and building features

[1737] Output: Generated 3D model

[1738] Step 5: The server sends the generated 3D model to the smart glasses

[1739] The server then transmits the generated 3D model to the smart glasses via wireless communication such as Wi-Fi or Bluetooth.

[1740] Input: Generated 3D model

[1741] Output: 3D model sent to smart glasses

[1742] Step 6: The smart glasses display the 3D model and the user experiences it.

[1743] The smart glasses then display the received 3D model in the user's field of vision, giving the user the experience of being transported back in time to Edo-period Tokyo.

[1744] Input: 3D model sent to smart glasses

[1745] Output: A 3D model displayed in the user's field of view.

[1746] In this way, the system allows users to experience past streetscapes in real time.

[1747] (Application example 1)

[1748] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1749] In modern brick-and-mortar stores, it is difficult for users to experience the streetscapes and buildings of the past. This limits opportunities to understand historical background and culture. In addition, there is a lack of ways to visually reproduce past scenery in tourist spots and historical places, making it difficult to attract users' interest.

[1750] The identification process by the identification processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means. In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscape on the smart glasses, and a means for allowing the user to experience past streetscapes in a real store. This allows the user to visually experience the appearance of past streetscapes and buildings in a real store, and to gain a deeper understanding of the historical background and culture.

[1751] "Smart glasses" are glasses-type devices that users can wear to provide augmented reality and display information.

[1752] An "antique map" is a map that shows geographical information from the past, recording historical geographical features and the layout of buildings.

[1753] "Old photographs" are photographs of landscapes, buildings, and people from the past, providing visual historical information.

[1754] "Means of analysis" refers to technology and equipment for digitally processing information from old maps and old photographs and extracting geographical and architectural features.

[1755] "Means of generation" refers to technology and equipment for creating 3D models of past cityscapes and buildings based on analyzed information.

[1756] "Display means" refers to the technology or devices for displaying the generated 3D model on the smart glasses and providing the user with a visual experience.

[1757] A "brick and mortar store" is a store that exists in a physical location and where users can visit in person to purchase goods or services.

[1758] "Means for experiencing streetscapes of the past" refers to technologies and devices that allow users to visually experience streetscapes and buildings of the past through smart glasses.

[1759] The system for implementing the present invention includes smart glasses, a server, an analysis module, a generation module, and a display module. Detailed processing of each module and the hardware and software used will be described below.

[1760] 1. System Programming

[1761] The system includes a program that works in conjunction with smart glasses to analyze information from old maps and old photographs, generate a 3D model of the cityscape of the past, and display it to the user.

[1762] 2. Program Processing Explanation

[1763] Hardware used:

[1764] Smart glasses (e.g. Microsoft HoloLens, Google Glass)

[1765] Server (a computer with high-performance data processing capabilities)

[1766] Software used:

[1767] OpenCV: Image analysis library

[1768] ARModule: Augmented reality display module

[1769] OldMapAnalyzer: Old map analysis module

[1770] ThreeDModelGenerator: 3D model generation module

[1771] Process flow:

[1772] 1. Loading old maps:

[1773] The server uses OpenCV to read image data from old maps and old photographs, thereby obtaining past geographical information and architectural features as digital data.

[1774] 2. Feature Extraction:

[1775] The server uses the OldMapAnalyzer module to extract geographic and architectural features from old maps and photographs, providing detailed information about past streetscapes.

[1776] 3. 3D model generation:

[1777] The server uses the ThreeDModelGenerator module to generate a 3D model based on the extracted features, which allows the past streetscape and buildings to be reproduced in three dimensions.

[1778] 4.Display:

[1779] The server transmits the generated 3D model to the smart glasses via ARModule and displays it to the user, allowing the user to visually experience the streetscape of the past in a brick-and-mortar store.

[1780] 3. Specific Examples

[1781] For example, when a user visits a historical tourist site, they can wear the smart glasses and experience the townscape of the Edo period, allowing them to enjoy sightseeing while comparing modern scenery with past scenery.

[1782] Example prompt:

[1783] "Please recreate the streets of Asakusa in the Edo period. Analyze old maps and old photographs, generate a 3D model and display it on the smart glasses."

[1784] In this way, users can enjoy shopping and sightseeing while comparing modern-day Asakusa with Asakusa of the Edo period.

[1785] The flow of the specific process in the application example 1 will be described with reference to FIG.

[1786] Step 1:

[1787] The server uses OpenCV to read image data of old maps and old photographs. It receives the image file path of the old map or old photograph as input and obtains image data as output. This image data is used for subsequent analysis processing.

[1788] Step 2:

[1789] The server uses the OldMapAnalyzer module to extract geographic and architectural features from the loaded image data. As input, it receives the image data acquired in step 1, and as output, it generates a dataset containing geographic and architectural features. This dataset will be the basis for reconstructing the streetscape of the past.

[1790] Step 3:

[1791] The server uses the ThreeDModelGenerator module to generate a 3D model based on the extracted features. It receives as input the dataset containing the geographical and architectural features generated in step 2, and generates as output 3D model data. This 3D model data is used to reproduce the appearance of past streets and buildings in three dimensions.

[1792] Step 4:

[1793] The server sends the generated 3D model data to the smart glasses through ARModule. It receives the 3D model data generated in step 3 as input and sends data to be displayed on the smart glasses as output. This data is used for the user to visually experience the past cityscape through the smart glasses.

[1794] Step 5:

[1795] The user wears the smart glasses and experiences the streetscape of the past in a real store. As input, the user receives 3D model data displayed on the smart glasses, and as output, the user visually experiences the streetscape and buildings of the past. Through this experience, the user can enjoy shopping and sightseeing while comparing the modern landscape with the landscape of the past.

[1796] Example 2

[1797] Next, a second embodiment of the second embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1798] In order to provide an experience that combines modern streetscapes with past streetscapes, technology is required to acquire appropriate historical data based on the user's current location, and analyze and generate it in real time. However, with conventional technology, there is no system that consistently acquires location information, analyzes historical data, and displays the generated streetscapes, making it difficult for users to smoothly experience past streetscapes.

[1799] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1800] In this invention, the server includes a means for acquiring an old map related to the vicinity of the location information, which is a map from a time before the present, and an old photograph related to the vicinity of the location information, based on the user's location information, a means for extracting geographical features and architectural features from the acquired old map and old photograph, a prompt sentence for instructing to generate a 3D model representing a landscape related to the vicinity of the location information, which is a landscape from a time before the present, based on the extracted geographical features and architectural features, a means for generating the 3D model based on a generative AI model, and a means for displaying an image obtained based on the generated 3D model on smart glasses worn by the user. This allows the user to smoothly enjoy an experience in which modern cityscapes and past cityscapes are blended in real time.

[1801] "Smart glasses" are wearable devices that, when worn by a user, can overlay digital information onto images from the real world.

[1802] “Location Information” means data that describes a User’s current geographic location, such as data obtained using GPS or other location measurement technologies.

[1803] A "server" is a computer system that processes and stores data via a network, and receives, analyzes, and generates information sent from a user's device.

[1804] An "antique map" is a map that shows geographical information from the past, and visually represents historical cityscapes and topography.

[1805] "Old photographs" are photographs taken in times past that record historical buildings and landscapes.

[1806] "Analysis" is the process of examining acquired data in detail and extracting useful information from it.

[1807] A "generative AI model" is an algorithm or system that uses artificial intelligence to generate new data and content, generating past streetscapes based on a prompt text.

[1808] A "prompt" is an instruction entered into a generative AI model to cause it to generate a specific output.

[1809] "Streetscape" refers to the layout and appearance of buildings, roads, and other structures in a particular area.

[1810] "Display" refers to the process of visually presenting digital information to a user by overlaying the information on the smart glasses display.

[1811] This invention is a system that allows a user to wear smart glasses and stand at a specific location, and generates an old streetscape based on information from old maps and old photographs of the location, providing an experience that blends the old streetscape with the real world. A specific embodiment of this system will be described below.

[1812] First, a user puts on the smart glasses and stands at a specific location, for example, in Asakusa, Tokyo. The smart glasses use the built-in GPS sensor to obtain the user's current location information. This location information is then sent to the server in real time.

[1813] The server receives the location information sent from the device. Based on the received location information, it uses geographic information services such as Google Maps API and OpenStreetMap API to obtain old maps and old photographs of the location. The obtained data is analyzed within the server, and geographical and architectural features are abstracted.

[1814] Based on the analyzed data, the server generates a prompt sentence to be input to a generative AI model (e.g., GPT-4 (registered trademark) or DALL-E). For example, the server creates a prompt sentence such as, "Based on geographical features and architectural features, please generate a 3D model that represents the scenery of the Edo period around the current location."

[1815] The generative AI model generates a 3D model representing the old scenery (cityscape) based on the prompt sent from the server. The generated data is returned to the server. The server receives the 3D model data representing the old scenery returned from the generative AI model and sends it to the terminal.

[1816] The device receives 3D model data representing old scenery sent from the server. The images obtained from the received data are superimposed on the real-world image on the smart glasses display, allowing users to enjoy an experience in which the real world and old townscapes are blended in real time.

[1817] As a concrete example, consider the case where a user stands in front of Kaminarimon Gate in Asakusa. When the user wears smart glasses and stands at a specific point in Asakusa, the smart glasses send the location information of that point to the server. The server uses the Google Maps API to obtain old maps and old photos of Asakusa, and inputs the following prompt text to the generative AI model:

[1818]

[1819] The generative AI model generates an image of Asakusa in the Edo period based on this prompt and returns the data to the server. The server then sends the generated data of Asakusa in the Edo period to the device. The device then overlays the received data with real-world images and displays them on the smart glasses display. The user can experience an image that combines modern Asakusa with Asakusa in the Edo period.

[1820] This system allows users to smoothly enjoy an experience in real time where modern cityscapes blend with historic streetscapes.

[1821] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1822] Step 1:

[1823] A user puts on the smart glasses and stands at a particular point.

[1824] Input: User's location

[1825] Output: Smart glasses get location information

[1826] Specific operation: A user wears the smart glasses and stands at a specific location, for example, in Asakusa, Tokyo. The smart glasses use the built-in GPS sensor to obtain the user's current location.

[1827] Step 2:

[1828] The device obtains location information and sends it to the server.

[1829] Input: Obtained location information

[1830] Output: Location sent to the server

[1831] How it works: The smart glasses transmit the acquired location information to the server in real time. The location information is sent to the server in the form of latitude and longitude.

[1832] Step 3:

[1833] The server receives the location information and retrieves the corresponding old maps and old photographs.

[1834] Input: Location sent

[1835] Output: Obtained old maps and old photographs

[1836] Specific operation: The server receives the location information sent from the device. Based on the received location information, it uses geographic information services such as Google Maps API and OpenStreetMap API to obtain old maps and old photos of that location.

[1837] Step 4:

[1838] The server analyzes the acquired data and inputs a prompt sentence along with the analysis results into the generative AI model.

[1839] Input: Obtained old maps and old photographs

[1840] Output: Prompt sentence to be input to the generative AI model

[1841] Specific operation: The server analyzes the acquired data of old maps and old photographs. Along with the analysis results, it generates a prompt sentence to be input to a generative AI model (e.g., GPT-4 or DALL-E). For example, it creates a prompt sentence such as, "Based on geographical features and architectural features, please generate a 3D model that represents the scenery of the Edo period around the current location."

[1842] Step 5:

[1843] The generative AI model generates a 3D model representing the old cityscape and returns the data to the server.

[1844] Input: prompt statement

[1845] Output: Generated 3D model data showing the old streetscape

[1846] Specific operation: The generative AI model generates old streetscapes based on the prompts sent from the server. The generated data is returned to the server.

[1847] Step 6:

[1848] The server transmits the generated data to the terminal.

[1849] Input: Generated old streetscape data

[1850] Output: Data sent to the terminal

[1851] Specific operation: The server receives the data of old cityscapes returned by the generative AI model and sends it to the device.

[1852] Step 7:

[1853] The data received by the device is overlaid on images of the real world and displayed.

[1854] Input: Received data of old streetscapes

[1855] Output: Fusion image displayed on the smart glasses display

[1856] Specific operation: The device receives data of old streetscapes sent from the server, and displays the received data on the smart glasses display, overlaying it with images of the real world.

[1857] Step 8:

[1858] Users will experience a fusion of the real world and old-fashioned streetscapes.

[1859] Input: Fusion image displayed on the smart glasses display

[1860] Output: The user's experience

[1861] Specific operation: Through the smart glasses, the user experiences images that combine modern cityscapes with those of the past. For example, the user can enjoy images of modern Asakusa overlapping with those of Asakusa in the Edo period.

[1862] In addition, the user's emotion may be recognized using an emotion engine described later, and the display content of the smart glasses may be adjusted based on the recognized emotion of the user. Specifically, the color tone of the scenery and the facial expressions of people included in the displayed image may be adjusted according to the user's emotion.

[1863] (Application example 2)

[1864] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[1865] Systems that allow users to visually experience a fusion of modern cityscapes and past cityscapes are extremely useful in the fields of tourism and education. However, with current technology, it is difficult for users to obtain, analyze, and visually display historical information about a specific location in real time when they stand there. In addition, technology that uses generative AI models to generate prompt sentences and display past cityscapes based on those sentences is still in its infancy. This poses the challenge of making it difficult for users to experience past cityscapes in real time.

[1866] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1867] In this invention, the server includes a means for linking with the smart glasses, a means for analyzing information on old maps and old photographs, a means for generating old streetscapes based on the analyzed information, a means for displaying the generated streetscapes on the smart glasses, a means for acquiring information on old maps and old photographs of a specific location when the user stands at the location, a means for generating a prompt sentence using a generative AI model based on the acquired information, and a means for visually displaying the old streetscape based on the generated prompt sentence. This enables the user to experience the past streetscape of a specific location in real time simply by standing at the location.

[1868] "Smart glasses" are wearable devices that allow users to overlay digital information onto the real world.

[1869] An "antique map" is a map that shows geographical information from the past, recording historical topography and the layout of buildings.

[1870] "Old photographs" are photographs of landscapes, buildings, and people from the past, and serve as visual records of historical information.

[1871] "Means of analysis" refers to techniques and methods for processing the information from acquired old maps and old photographs as data and extracting geographical and architectural features.

[1872] "Means of generation" refers to the techniques and methods for recreating past streetscapes and landscapes as digital data based on analyzed information.

[1873] "Display means" refers to the technology or method for displaying the generated past cityscapes and scenery on the smart glasses.

[1874] A "particular location" is a location where a user is standing and about which historical information relating to that location is retrieved.

[1875] A "generative AI model" is a model that uses artificial intelligence to analyze data and generate output for a specific purpose.

[1876] A "prompt sentence" is an instruction sentence output by a generative AI model, and is text generated based on specific information.

[1877] "Means for visual display" refers to techniques or methods for displaying the generated past cityscapes or scenery so that the user can visually experience them.

[1878] The system for implementing the present invention includes smart glasses, a server, and a generative AI model. A specific embodiment of the system is described below.

[1879] First, when a user wears the smart glasses and stands at a specific location, the smart glasses send the location information of that location to the server. Based on the location information, the server retrieves information on old maps and old photographs related to that location from a database that stores past geographical information and historical photographs.

[1880] Next, the server analyzes the information on the acquired old maps and old photographs. Image processing techniques are used for the analysis to extract geographical and architectural features. Specifically, image processing libraries such as OpenCV are used to extract feature points from maps and photographs.

[1881] Based on the analyzed information, the server generates a prompt sentence using a generative AI model. For the generative AI model, a natural language generation model such as GPT-3 is used. The generated prompt sentence is an instruction sentence for reproducing the past streetscape.

[1882] Based on the generated prompt, the server generates digital data of the streetscape of the past. This generation is done using 3D modeling software and rendering engines, such as Blender and Unity.

[1883] The generated past streetscapes are sent to the smart glasses and displayed in the user's field of view, allowing the user to have a visual experience that blends the real world with the past streetscapes.

[1884] As a concrete example, when a user stands at Tokyo Station, the server retrieves old photos of Tokyo Station, analyzes them, and generates a prompt. The generated prompt is "Please show me the old streetscape of Tokyo Station." Based on this prompt, the server generates an old streetscape of Tokyo Station and displays it on the smart glasses. The user can enjoy walking around while comparing the modern Tokyo Station with the old streetscape of Tokyo Station.

[1885] In this way, the present invention enables a user to experience a past streetscape of a particular location in real time, simply by standing at that location.

[1886] The flow of the specific process in the application example 2 will be described with reference to FIG.

[1887] Step 1:

[1888] A user puts on the smart glasses and stands at a particular point.

[1889] Input: User's location

[1890] Output: Location information is sent from the smart glasses to the server.

[1891] Specific operation: The GPS function of the smart glasses obtains the user's current location and sends that location information to the server.

[1892] Step 2:

[1893] Based on the location information, the server retrieves information on old maps and old photographs related to that location from a database.

[1894] Input:Location

[1895] Output: Information on old maps and old photographs

[1896] Specific operation: The server queries the database using the location information as a key and retrieves the relevant data on old maps and old photographs.

[1897] Step 3:

[1898] The server analyzes the information on old maps and old photographs it has acquired.

[1899] Input: Information on old maps and old photographs

[1900] Output: Geographical and architectural features

[1901] Specific operation: The server uses image processing libraries such as OpenCV to extract feature points from old maps and old photographs.

[1902] Step 4:

[1903] The server generates a prompt sentence based on the parsed information using a generative AI model.

[1904] Input: Geographical and architectural features

[1905] Output: Prompt statement

[1906] Specific operation: The server inputs feature data into a generative AI model such as GPT-3 and generates prompt sentences to recreate past cityscapes.

[1907] Step 5:

[1908] The server generates past streetscapes as digital data based on the generated prompt sentence.

[1909] Input: Prompt statement

[1910] Output: Digital data of past streetscapes

[1911] Specific operation: The server uses 3D modeling software such as Blender or Unity to generate a 3D...

Claims

1. A means for acquiring, based on the user's location information, an old map relating to the vicinity of the location information, which is an older map than the present, and an old photograph relating to the vicinity of the location information, which is an older photograph than the present; a means for generating the three-dimensional model based on a prompt sentence instructing to generate a three-dimensional model representing a landscape from an earlier time period and related to the surroundings of the location information, based on the old map and the old photograph, and based on a generating AI model; a means for displaying an image obtained based on the generated three-dimensional model on smart glasses worn by a user; A system including:

2. A means for recognizing a user's emotion; and means for adjusting display content of the smart glasses based on the recognized emotion of the user.

3. 3. The system according to claim 2, wherein said display content adjusting means adjusts the color tone of the scenery and the facial expressions of people included in the displayed image in accordance with the emotion of the user.

4. The method further includes means for extracting geographical features and architectural features from the acquired historical maps and photographs; The system of claim 1, wherein the means for generating the three-dimensional model generates the three-dimensional model based on a prompt sentence instructing the generation of a three-dimensional model representing a landscape relating to the surroundings of the location information that is a landscape from a past time before the present, based on the extracted geographical features and architectural features, and based on a generative AI model.

Images