System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to upload a plurality of images via a device; means for a server to store the images in cloud storage; means for a generative algorithm to extract and analyze metadata of the images; means for an interactive algorithm to ask users for detailed information and store their answers; means for automatically generating detailed records based on the generative algorithm and user answers; and means for allowing the automatically generated records to be shared between users.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the widespread use of digital cameras and smartphones, many users take a large number of photos while traveling. However, organizing these photos and creating travel journals takes time and effort, and many users find it tedious and difficult. Furthermore, it can be difficult to share detailed travel information and memories with other users or to look back on them for themselves. For this reason, there is a demand for a system that allows users to easily organize travel photos and automatically generate detailed travel journals. [Means for solving the problem]

[0005] The present invention solves the above problem by providing a system including means for a user to upload multiple images via a device, means for a server to store the images in cloud storage, means for a generation algorithm to extract and analyze metadata from the images, means for an interactive algorithm to ask the user detailed information and store the answers, means for automatically generating detailed records based on the generation algorithm and the user's answers, and means for enabling the automatically generated records to be shared among users.

[0006] Furthermore, in order to provide a virtual experience, by providing a means for enabling a virtual travel experience based on the automatically generated record that has been made public, other users can also obtain information and experiences about new travel destinations through the travel diary. Also, by providing a means for the user to skip questions when the interactive algorithm presents questions to the user, the user can proceed with the answers at their own pace, improving the usability of the system.

[0007] "User" refers to any individual or entity that uses the System to upload photos and generate travelogues.

[0008] "Device" refers to an electronic device, such as a computer, smartphone, or tablet, that a User uses to connect to the Internet and access the System.

[0009] "Image" means a digital photograph or scanned image that you take using your Device and upload to the System.

[0010] "Server" refers to a computer system that stores and analyzes images, manages generated travelogues, etc.

[0011] "Cloud storage" refers to a remote server system for storing and managing data over the Internet.

[0012] "Metadata" refers to information associated with an image (such as the date and time the image was taken or GPS coordinates).

[0013] "Generation algorithm" refers to a computer program for extracting and analyzing metadata from images.

[0014] "Interactive algorithm" refers to an artificial intelligence program that presents questions to users and collects their answers.

[0015] "Detailed information" refers to additional information such as the user's travel stories and impressions.

[0016] "Record" refers to the text and content generated by generative and interactive algorithms that summarizes travel details.

[0017] "Sharing" refers to making a generated record available for viewing by other users.

[0018] "Virtual travel experience" refers to other users virtually experiencing a trip through a shared recording.

[0019] "Means to allow questions to be skipped" refers to a function that allows the user to proceed to the next question presented to the user by the interactive algorithm without responding. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] This invention relates to a system that allows a user to upload multiple photos taken during a trip, analyzes the date, time, location, and detailed information from those photos, and automatically generates a travel journal. Specific embodiments for implementing this system and their program processing are described below.

[0042] User Registration and Login

[0043] A user accesses the system using a terminal and first registers, entering information such as name, email address, and password into the registration form, which the terminal then sends to the server, which receives the information and creates a new user account in the database.

[0044] When logging in, the user enters their email address and password from their device and sends them to the server. The server then refers to the database for authentication, and if successful, starts a session.

[0045] Photo upload

[0046] The user uploads multiple photos taken during their trip from their device. The device sends a request to upload the selected photos to the server, and the server stores the received photos in cloud storage. The server records the storage location of each photo in a database.

[0047] Photo analysis

[0048] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts and analyzes the photo's metadata (such as the date and time of the photo's capture, GPS information, etc.). The extracted metadata is then stored in a database by the server.

[0049] Gathering more information with conversational AI

[0050] The server uses an interactive algorithm to generate questions based on the parsed metadata. For example, questions like "Where was this photo taken?" or "What experience did you have here?" are sent to the user's device, and the user answers them. The user can also skip questions. The device sends the user's answers to the server, which stores them in a database.

[0051] Automatic travel report generation

[0052] The server selects an appropriate travelogue template based on the collected information (analysis metadata and user responses) and automatically generates a detailed travelogue, which includes text, photos, anecdotes, etc. The generated travelogue is then delivered to the user's device for viewing.

[0053] Information sharing and virtual travel experiences

[0054] Users can set their travel journals to be public or private using their devices. The setting information is sent to the server, which stores it in a database. Travel journals set to public can be viewed by other users. The server generates a virtual travel experience based on the published travel journal and provides it to other users, along with detailed information about related places and activities.

[0055] These processes allow users to easily organize their travel photos, automatically generate detailed and engaging travel journals, and share them with other users to enjoy virtual travel experiences.

[0056] The processing flow will be explained below.

[0057] Step 1:

[0058] A user accesses the system using a terminal and enters the required information, such as name, email address, and password, into the new registration form.

[0059] Step 2:

[0060] The device sends the user's input information to the server and makes a registration request.

[0061] Step 3:

[0062] The server takes the information you enter and creates a new user account in its database.

[0063] Step 4:

[0064] From next time onwards, the user will enter their email address and password into the login form on their device to request a login.

[0065] Step 5:

[0066] The terminal sends authentication information to the server, and the server performs authentication by referencing a database.

[0067] Step 6:

[0068] If the server successfully authenticates the user, it starts a session for the user and grants access rights.

[0069] Step 7:

[0070] The user uses the device to select multiple photos taken during the trip and submits an upload request.

[0071] Step 8:

[0072] The device transmits the selected photo data to the server, and the server stores the photos in cloud storage.

[0073] Step 9:

[0074] The server records the location of each photo in a database.

[0075] Step 10:

[0076] The server sequentially passes the photos stored in cloud storage to the generation algorithm.

[0077] Step 11:

[0078] The generation algorithm extracts and analyzes the photo's metadata (such as the date and time the photo was taken and GPS information).

[0079] Step 12:

[0080] The server stores the extracted metadata in a database and organizes the parsed information.

[0081] Step 13:

[0082] The server uses an interactive algorithm to generate questions for the user based on the parsed metadata.

[0083] Step 14:

[0084] The interactive algorithm generates questions and sends them to the user's device.

[0085] Step 15:

[0086] The user uses the device to answer questions posed by the conversational algorithm or to skip questions.

[0087] Step 16:

[0088] The device sends the user's answers to the server, which stores them in a database.

[0089] Step 17:

[0090] The server selects an appropriate travelogue template based on the information it collects (metadata and user responses).

[0091] Step 18:

[0092] The server fills in the information in the selected template and automatically generates a detailed travelogue.

[0093] Step 19:

[0094] The server provides the generated travelogue to the user's terminal, and the user views the travelogue.

[0095] Step 20:

[0096] Users can use their devices to set their travel journals to be public or private.

[0097] Step 21:

[0098] The terminal sends the configuration information to the server, which stores it in a database.

[0099] Step 22:

[0100] Other users can use their devices to view your published travelogue.

[0101] Step 23:

[0102] The server generates a virtual travel experience based on the published travelogue and provides it to other users.

[0103] Step 24:

[0104] The server also provides detailed information about related places and activities to support virtual travel.

[0105] These steps allow users to easily organize their travel photos, automatically generate and share detailed travel journals, and share their travel journals with other users to enjoy virtual travel experiences.

[0106] Example 1

[0107] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0108] Efficiently organizing the many images taken during a trip and generating a detailed travelogue based on them is a time-consuming and labor-intensive process. There is also a need for a method to share travelogues with other users and provide them with virtual travel experiences. However, existing systems lack the functionality to allow users to easily analyze image metadata and automatically generate detailed travelogues. Furthermore, they lack an interactive algorithm that generates questions based on user answers, and an effective means to provide virtual travel experiences.

[0109] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0110] In this invention, the server includes a means for users to upload multiple images via their terminals, a means for the server to store the images in cloud storage, a means for a generation algorithm to extract and analyze metadata from the images, a means for an interactive algorithm to ask the user detailed information and store the answers, a means for automatically generating a detailed travelogue based on the generation algorithm and the user's answers, and a means for enabling the automatically generated travelogue to be shared among users. This allows users to easily upload images taken during their trip and automatically generate and share a detailed travelogue based on the images. A virtual travel experience is also provided, allowing other users to share their travelogues and enjoy virtual experiences.

[0111] "User" means the person who uses the system to upload images or input information.

[0112] A "terminal" is an electronic device such as a computer or smartphone used by a user.

[0113] "Images" are photo files taken by users during their travels and uploaded to the system.

[0114] The "server" refers to a central management system that stores images and analyzes metadata.

[0115] "Cloud storage" is an external service for storing and managing data via the Internet.

[0116] "Generation algorithm" refers to software that extracts and analyzes image metadata.

[0117] "Metadata" is additional information including the date and time an image was taken and GPS information.

[0118] An "interactive algorithm" is software that generates questions to gather more information from users and stores their answers.

[0119] A "travel journal" is a written record generated based on images and information provided by the user.

[0120] "Virtual Travel Experience" is a feature that allows other users to virtually experience travel based on automatically generated travelogues.

[0121] This invention relates to a system that allows a user to upload multiple images taken during a trip, analyzes the date, time, location, and detailed information from those images, and automatically generates a travelogue. Specific embodiments for implementing this system are described below.

[0122] User Registration and Login

[0123] A user accesses the system using a terminal and first registers. The user enters their name, email address, and password into a registration form, and the terminal sends this information to the server. The server receives the entered information and creates a new user account in the database. When the user logs in, they enter their email address and password and send them to the server. The server refers to the database to authenticate them, and if successful, starts a session.

[0124] Specific behavior:

[0125] The user fills in the required information in the registration form and clicks the "Register" button.

[0126] The device sends the input to the server.

[0127] The server creates a new user account in the database.

[0128] Photo upload

[0129] The user uploads multiple photos taken during their trip from their device. The device sends a request to upload the selected photos to the server, and the server stores the received photos in cloud storage such as Google Cloud Storage. The server records the storage location of each photo in a database.

[0130] Specific behavior:

[0131] The user selects multiple photos from a file folder and clicks the "Upload" button.

[0132] The terminal transmits the selected photo file to the server.

[0133] The server stores the photo in Google Cloud Storage and records the URL in a database.

[0134] Photo analysis

[0135] The server sequentially retrieves photos stored in cloud storage and extracts the photo metadata (date and time of shooting, GPS information, etc.) using image analysis libraries such as OpenCV and TensorFlow. The extracted metadata is then stored in a database by the server.

[0136] Specific behavior:

[0137] The server retrieves the photos from the cloud storage.

[0138] The server uses an image analysis library to extract the metadata.

[0139] The extracted metadata is stored in a database.

[0140] Gathering more information with conversational AI

[0141] The server uses an interactive algorithm to generate questions based on the parsed metadata, sends the generated questions to the user's device, and the user can answer or skip the questions. The device then sends the user's answers to the server, which stores them in a database.

[0142] Specific behavior:

[0143] The server generates questions based on the metadata.

[0144] Example questions: "Where was this photo taken?"

[0145] The device displays a question to the user.

[0146] The user enters the answer and the device sends the answer to the server.

[0147] Automatic travel report generation

[0148] The server selects an appropriate travelogue template based on the collected information (analysis metadata and user responses), and automatically generates a detailed travelogue using a generative AI model such as GPT-3. The generated travelogue is then sent to the user's device for viewing.

[0149] Example prompt sentence:

[0150] "These are travelogues to Tokyo. Please generate a detailed travelogue based on the information below."

[0151] Specific behavior:

[0152] The server sends a prompt to the generative AI model.

[0153] The generated travelogue is sent to the user's device.

[0154] Information sharing and virtual travel experiences

[0155] Users set their travel journals to be public or private and send the setting information to the server. Travel journals that are public can be viewed by other users. The server generates a virtual travel experience based on the published travel journal and provides it to other users.

[0156] Specific behavior:

[0157] Users can set their travel journals to be public or private.

[0158] The server acquires publicly set travelogues and generates a virtual travel experience.

[0159] By implementing this system, users can easily organize their travel photos, automatically generate detailed and attractive travel journals, and share them with other users to enjoy virtual travel experiences.

[0160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0161] Step 1: User Registration

[0162] The user enters their name, email address, and password in plain text.

[0163] The device sends this input information to the server in JSON format.

[0164] The server stores the received information in its database as a new user account.

[0165] Enter your name, email address, and password.

[0166] Data processing: Convert input information into JSON format.

[0167] Output: A new user account is created in the database.

[0168] Specific behavior:

[0169] The user enters "Yamada Taro", "taro@example.com", and a password in the registration form and clicks the "Register" button.

[0170] The device sends this information to the server in JSON format.

[0171] The server receives the information and creates a new user account in the database.

[0172] Step 2: Log in

[0173] The user enters their email address and password in plain text.

[0174] The device sends this input information to the server in JSON format.

[0175] The server refers to the database and performs authentication. If authentication is successful, a session is started.

[0176] Enter your email address and password.

[0177] Data calculation: Matching input information with existing information in a database.

[0178] Output: A session ID is generated.

[0179] Specific behavior:

[0180] The user enters "taro@example.com" and their password in the login form and clicks the "Login" button.

[0181] The device sends this information to the server in JSON format.

[0182] The server queries the database, and since authentication is successful, generates a session ID.

[0183] Step 3: Upload your photos

[0184] The user selects multiple photos taken during their trip from their device. The selection is made in file format.

[0185] The device sends the selected photo file to the server via an HTTP POST request.

[0186] The server stores the received photos in Google Cloud Storage and records their URLs in the database.

[0187] Input: Travel photo files.

[0188] Data processing: Convert the photo file into HTTP POST request format.

[0189] Output: The photo is saved to Google Cloud Storage and its URL is recorded in the database.

[0190] Specific behavior:

[0191] The user selects "IMG_001.jpg" to "IMG_010.jpg" from the file folder and clicks the "Upload" button.

[0192] The device sends the selected photo file to the server via an HTTP POST request.

[0193] The server stores the received photos in Google Cloud Storage and records their URLs in the database.

[0194] Step 4: Photo analysis

[0195] The server retrieves photos sequentially from Google Cloud Storage.

[0196] The server uses image analysis libraries such as OpenCV and TensorFlow to extract photo metadata (such as the date and time the photo was taken, GPS information, etc.).

[0197] The server stores the extracted metadata in a database.

[0198] Input: Google Cloud Storage photo URL.

[0199] Data Computation: Extraction of metadata using image analysis libraries.

[0200] Output: The extracted metadata is stored in a database.

[0201] Specific behavior:

[0202] The server retrieves "IMG_001.jpg" from Google Cloud Storage.

[0203] The server analyzes the EXIF ​​information of the photo using OpenCV and extracts the date and time of the photo and GPS information.

[0204] The extracted metadata is stored in a database.

[0205] Step 5: Gathering more information with conversational AI

[0206] The server generates questions based on the parsed metadata, using a generative AI model (e.g., GPT-3).

[0207] The device displays the generated question to the user, who can then enter an answer or skip.

[0208] The device sends the user's answers to the server, which stores them in a database.

[0209] Input: Metadata.

[0210] Data Computing: Generate questions and store answers using generative AI models.

[0211] Output: The generated questions and the user's answers are stored in a database.

[0212] Specific behavior:

[0213] Based on the metadata, the server sends the question "Where was this photo taken?" as a prompt to the generative AI model and receives the question as an answer.

[0214] The device displays a question to the user.

[0215] The user answers "Tokyo Tower," and the device sends the answer to the server.

[0216] The server stores the answer in a database.

[0217] Step 6: Automatic travel journal generation

[0218] The server selects an appropriate travelogue template based on the collected metadata and the user's answers.

[0219] The server automatically generates a detailed travelogue using a generative AI model (e.g., GPT-3).

[0220] The server sends the generated travelogue to the user's terminal.

[0221] Input: Metadata, user answers.

[0222] Data computation: Generating travel journals using generative AI models.

[0223] Output: The automatically generated travelogue is sent to the user's device.

[0224] Specific behavior:

[0225] The server converts the metadata and answers into prompts and sends them to the generative AI model.

[0226] The generated travelogue is sent to the user's device and can be viewed by the user.

[0227] Step 7: Information sharing and virtual travel experiences

[0228] The user sets the travel journal to be public or private and sends it from the device to the server.

[0229] The server stores the configuration information in a database.

[0230] The server generates a virtual travel experience based on the published travelogue and provides it to other users.

[0231] Input: Public / Private settings.

[0232] Data calculation: Converting published travel journal data into virtual experience content.

[0233] Output: The publicly set travelogue is shared with other users, providing them with a virtual travel experience.

[0234] Specific behavior:

[0235] The user sets whether their travel journal is public or private, and sends the setting information from their device to the server.

[0236] The server stores the configuration information in a database.

[0237] The server acquires the published travelogue data, generates a virtual travel experience, and provides it to other users.

[0238] The above is the specific processing flow of this system and the operation at each step.

[0239] (Application example 1)

[0240] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0241] In today's world, many users find it time-consuming to simply organize photos taken during a trip and record their experiences in a detailed travel journal. It's also difficult to share those travel journals with others or provide virtual travel experiences. This often results in valuable travel experiences remaining buried. The present invention aims to solve these problems by providing a system that automatically and efficiently generates travel journals based on photos taken by users and allows them to publish and share them in a virtual store.

[0242] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0243] In this invention, the server includes: a means for a user to upload multiple images via a terminal; a means for the server to store the images in cloud storage; a means for a generation algorithm to extract and analyze metadata from the images; a means for an interactive algorithm to ask the user detailed information and store the answers; a means for automatically generating detailed records based on the generation algorithm and the user's answers; a means for enabling the automatically generated records to be shared among users; a means for providing a virtual experience based on the collected information and the generated records; and a means for publishing the generated records in a virtual store. This allows users to efficiently organize their travel photos, automatically generate and publish attractive travel journals, and share and enjoy them with other users.

[0244] "User" means any person or entity that accesses the system, uploads multiple images, and inputs information to generate detailed records from those images.

[0245] "Terminal" means the hardware device used by a User to access the System and upload images or enter information.

[0246] "Server" refers to the back-end computing resources used to store images received from users and run generative and interactive algorithms.

[0247] "Cloud storage" refers to an external storage system that allows a server to safely store image data received via the Internet.

[0248] "Generation Algorithm" refers to the software process used to extract and analyze metadata from uploaded images and automatically generate detailed records.

[0249] "Metadata" refers to data that contains additional information specific to an image, such as the date and time the image was taken or GPS information.

[0250] "Interactive algorithm" refers to the software process by which a system generates questions to gather more information from users and stores their answers.

[0251] "Detailed Records" refers to travel journals and other documents that are automatically generated based on images and information provided by users.

[0252] "Virtual Store" refers to an online platform for publishing and sharing detailed user-generated records with other users.

[0253] A "virtual experience" is an experience that allows other users to take a virtual journey based on a publicly detailed record.

[0254] This invention relates to a system that allows users to upload multiple images taken during their trip, analyzes the date, time, location, and detailed information from those images, and automatically generates a detailed record. Specific embodiments for implementing this system and their program processing are described below.

[0255] User Registration and Login

[0256] A user accesses the system using a terminal and first registers. They enter information such as their name, email address, and password into the registration form, and the terminal sends this information to the server. The server receives the entered information and creates a new user account in its database. When logging in, the user enters their email address and password from the terminal and sends them to the server. The server refers to the database to perform authentication, and if successful, starts a session.

[0257] Photo upload

[0258] The user uploads multiple photos taken during their trip from their device. The device sends a request to upload the selected photos to the server, which then stores the photos in cloud storage. The server records the location of each photo in a database.

[0259] Photo analysis

[0260] The server sequentially passes photos stored in the cloud storage to a generation algorithm, which extracts and analyzes the photo's metadata (such as the date and time of the photo's capture, GPS information, etc.) The extracted metadata is then stored in a database by the server.

[0261] Gathering more information with conversational AI

[0262] The server uses an interactive algorithm to generate questions based on the parsed metadata. For example, questions like "Where was this photo taken?" or "What experience did you have here?" are sent to the user's device, and the user answers them. The user can also skip questions. The device sends the user's answers to the server, which stores them in a database.

[0263] Examples of specific prompts include:

[0264] 1. Question: "Where was this photo taken?"

[0265] User Answer: "This photo was taken at Kiyomizu-dera Temple in Kyoto."

[0266] 2. Question: "What was your experience here?"

[0267] User Answer: "I enjoyed visiting tourist spots and eating my way around."

[0268] Detailed automatic recording

[0269] The server selects an appropriate template based on the collected information (analysis metadata and user responses) and automatically generates a detailed record, which includes text, photos, anecdotes, etc. The generated record is then delivered to the user's device for viewing.

[0270] Virtual Experience and Information Sharing

[0271] Users can use their devices to set their detailed records to public or private. The setting information is sent to the server, which stores it in a database. Publicly set detailed records can be viewed by other users. The server generates a virtual experience based on the publicly set detailed records and provides it to other users, along with detailed information about related locations and activities.

[0272] The system is implemented using the following hardware and software:

[0273] Server: Cloud server, database (e.g., Amazon RDS)

[0274] Image storage: Cloud storage (e.g. AWS S3)

[0275] Generative AI models: Libraries for running generative algorithms, such as TensorFlow and PyTorch

[0276] This allows users to efficiently organize their travel photos, automatically generate and publish compelling travel journals, and share and enjoy them with other users.

[0277] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0278] Step 1: User registration and login

[0279] A user accesses the system using a terminal, enters their name, email address, and password, and submits it. The server receives this information and creates a new user account in the database. When logging in, the user enters their email address and password. The server uses this information to look up their database and authenticates them, and if successful, starts a session.

[0280] Input: User's name, email address, password

[0281] Output: New user account created and authenticated.

[0282] Step 2: Upload a photo

[0283] Users select multiple travel photos from their devices and send an upload request to the server, which then stores the photos in cloud storage and records their location in a database.

[0284] Input: Multiple photos taken by the user

[0285] Output: Save photos to cloud storage and record their location in a database

[0286] Step 3: Photo analysis

[0287] The server passes the photos stored in the cloud storage to a generation algorithm, which extracts and analyzes the photo's metadata (date and time of the photo, GPS information), which is then stored in a database by the server.

[0288] Input: Photos stored in cloud storage

[0289] Output: Extracted metadata stored in a database

[0290] Step 4: Gathering more information with conversational AI

[0291] The server uses an interactive algorithm to generate questions based on the parsed metadata. The generated questions are sent to the user's device, and the user answers them. The device then sends the user's answers to the server, which stores them in a database.

[0292] Input: extracted metadata, user answers

[0293] Output: Generated questions, saving user answers to database

[0294] Examples of specific prompts include:

[0295] 1. Question: "Where was this photo taken?"

[0296] User Answer: "This photo was taken at Kiyomizu-dera Temple in Kyoto."

[0297] 2. Question: "What was your experience here?"

[0298] User Answer: "I enjoyed visiting tourist spots and eating my way around."

[0299] Step 5: Automatically generate detailed records

[0300] The server automatically generates a detailed record based on the generation algorithm and the user's answers. The generated record includes text, photos, anecdotes, etc. The server then provides the generated record to the user's device.

[0301] Input: Analysis metadata, user answers

[0302] Output: Auto-generated detailed transcript

[0303] Step 6: Virtual Experience and Information Sharing

[0304] Users can use their devices to set their detailed records to public or private. The server stores this setting information in a database. Publicly set detailed records can be viewed by other users. The server generates a virtual experience based on the publicly set detailed records and provides it to other users.

[0305] Input: User's public / private settings

[0306] Output: Publishing detailed records and generating virtual experiences

[0307] These processing flows allow users to efficiently organize photos taken during their trip, automatically generate and publish detailed and engaging travel journals, and share their virtual experiences with other users.

[0308] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0309] The present invention relates to a system that allows users to upload multiple photos taken during their trip, analyzes the date, time, location, and detailed information from those photos, and automatically generates a travel diary. Furthermore, by combining this system with an emotion engine, the system can recognize the user's emotions and reflect them in the travel diary. Specific embodiments of the present invention and the processing of the program are described below.

[0310] User Registration and Login

[0311] A user accesses the system using a terminal and enters the required information such as name, email address, password, etc. into a new registration form. The terminal sends this information to the server, which receives it and creates a new user account in the database.

[0312] When logging in, the user enters their email address and password from their device and sends them to the server. The server then refers to the database for authentication, and if authentication is successful, a session begins.

[0313] Photo upload

[0314] The user selects multiple photos taken during their trip using their device and makes an upload request. The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[0315] Photo analysis

[0316] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts metadata from the photos and analyzes information such as date, time, and location. The extracted metadata is then stored in a database by the server.

[0317] Gathering more information with conversational AI

[0318] The server uses an interactive algorithm to generate questions for the user based on the parsed metadata. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. The user can also skip questions. The device sends the answers to the server, which stores them in a database.

[0319] Emotional regulation by emotion engine

[0320] The emotion engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, when a user inputs an emotion expression such as "I had fun" or "I'm a little tired," the emotion engine analyzes the information and generates emotion data. The emotion data is then sent to the server and stored in a database.

[0321] Automatic travel report generation

[0322] The server selects an appropriate travelogue template based on the collected metadata, user responses, and emotion data. A detailed travelogue is automatically generated based on the selected template. This travelogue includes text, photos, anecdotes, and the user's emotions. The generated travelogue is then provided to the user's device for viewing.

[0323] Information sharing and virtual travel experiences

[0324] Users can set their travel journals to be public or private using their devices. The setting information is sent to the server and stored in a database. Travel journals set to public can be viewed by other users. The server generates a virtual travel experience based on the published travel journal and provides other users with detailed information on related places and activities.

[0325] For example, if a user uploads photos taken during a trip and inputs the emotion "It was fun" through the emotion engine, the system will analyze this information and reflect it in the travelogue as a "fun memory." In addition, when other users view this travelogue, content will be provided that allows them to feel a "fun atmosphere" as if they are traveling virtually.

[0326] This allows the system of the present invention to not only allow users to easily organize their travel photos and automatically generate detailed travel journals, but also to create records that reflect the user's emotions and share them with other users. Furthermore, other users can gain information and experiences about new travel destinations through virtual travel experiences.

[0327] The processing flow will be explained below.

[0328] Step 1:

[0329] A user accesses the system using a terminal and enters the required information, such as name, email address, and password, into the new registration form.

[0330] Step 2:

[0331] The terminal sends the entered user information to the server, which then creates a new user account in the database.

[0332] Step 3:

[0333] The user uses a device to enter their email address and password into the login form to request a login.

[0334] Step 4:

[0335] The terminal sends authentication information to the server, which then refers to the database to authenticate the user. If authentication is successful, the server starts a session for the user.

[0336] Step 5:

[0337] A user uses a device to select multiple travel photos and make an upload request.

[0338] Step 6:

[0339] The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[0340] Step 7:

[0341] The server sequentially passes the photos stored in cloud storage to the generation algorithm.

[0342] Step 8:

[0343] The generation algorithm extracts and analyzes the metadata of the photo (such as the date and time of the photo being taken, GPS information, etc.), and the extracted metadata is stored in a database by the server.

[0344] Step 9:

[0345] The server uses an interactive algorithm to generate questions for the user based on the parsed metadata.

[0346] Step 10:

[0347] The conversational algorithm generates questions and sends them to the user's device, such as "Where was this photo taken?" or "What kind of experience did you have here?"

[0348] Step 11:

[0349] The user uses the device to answer questions posed by the conversational algorithm or to skip questions.

[0350] Step 12:

[0351] The device sends the user's answers to the server, which stores them in a database.

[0352] Step 13:

[0353] The conversational algorithm uses an emotion engine to recognize the user's emotions. For example, if the user answers "I had fun" or "I'm a little tired," the emotion will be recognized.

[0354] Step 14:

[0355] The emotion engine generates the recognized emotion data and sends it to the server, which stores the emotion data in a database.

[0356] Step 15:

[0357] The server selects an appropriate travel diary template based on the collected metadata, user responses, and sentiment data.

[0358] Step 16:

[0359] The server automatically generates a detailed travelogue by filling in the selected template with information, including text, photos, anecdotes, and the user's emotions.

[0360] Step 17:

[0361] The server provides the generated travelogue to the user's terminal, and the user views the travelogue.

[0362] Step 18:

[0363] Users can use their devices to set their travel journals to be public or private.

[0364] Step 19:

[0365] The terminal sends the configuration information to the server, which stores the configuration information in a database.

[0366] Step 20:

[0367] Other users can use their devices to view your published travelogue.

[0368] Step 21:

[0369] The server generates a virtual travel experience based on the published travelogue. Emotional data is also taken into consideration, and content adapted to the user's emotions is provided in the virtual travel experience.

[0370] Step 22:

[0371] The server provides detailed information about places and activities related to the virtual travel experience, and helps other users gain information and experiences about new travel destinations through the travelogue.

[0372] These steps allow users to easily organize their travel photos, automatically generate and share detailed and emotional travel journals, and allow other users to experience virtual travel through the journals.

[0373] Example 2

[0374] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0375] Conventional methods require users to organize multiple photos taken during a trip and automatically generate a travelogue by analyzing the date, time, location, and detailed information from those photos. This requires a lot of effort and effort. It is also difficult to create a travelogue that reflects emotions, and it is not easy to share or provide a virtual travel experience. There is a need for a system that solves these problems and enables efficient automatic generation and sharing of travelogues that reflect emotions.

[0376] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0377] In this invention, the server includes means for a user to upload multiple images via a device, means for the server to store the images in a cloud storage device, means for a generation algorithm to extract and analyze metadata for the images, means for an interactive algorithm to ask the user detailed information and store the responses, means for an emotion recognition engine to recognize the user's emotions and analyze and store the emotion data, means for automatically generating detailed travelogues based on the generation algorithm, the user's responses, and the emotion data, and means for enabling the automatically generated travelogues to be shared among users, thereby enabling users to efficiently organize photos taken during their trips, automatically generate detailed travelogues, and create travelogues that reflect their emotions and share them with other users.

[0378] "User" refers to any individual or entity that uses the System to generate travel journals and upload photos.

[0379] "Device" refers to the hardware used by a user, such as a computer, smartphone, or tablet.

[0380] "Images" refers to photographic data taken by users during their travels and uploaded to the system.

[0381] "Server" refers to a computer system that controls the entire system and exchanges data with users.

[0382] "Cloud storage" refers to a storage service for storing data via the Internet.

[0383] "Generation algorithm" refers to a program for extracting and analyzing image metadata.

[0384] "Metadata" refers to additional information contained in an image, such as the date, time, location, and shooting conditions.

[0385] An "interactive algorithm" is a program designed to ask users detailed questions and collect their answers.

[0386] "Emotion recognition engine" refers to a system for analyzing and storing user emotional data.

[0387] "Emotional data" refers to information analyzed by an emotion recognition engine based on emotional expressions entered by the user.

[0388] "Records" refers to detailed travel journals that are automatically generated based on generative algorithms, user responses, and sentiment data.

[0389] "Virtual travel experience" refers to content that allows other users to experience the atmosphere and information of a trip based on publicly available records.

[0390] "Publicity Settings" refers to the settings options that allow a User to make a Recording that the User generates available to other Users.

[0391] The present invention relates to a system that allows users to upload multiple photos taken during their trip, analyzes the date, time, location, and detailed information from those photos, and automatically generates a travel diary. Furthermore, by combining this system with an emotion recognition engine, the system can recognize the user's emotions and reflect them in the travel diary. Specific embodiments of the present invention and the processing of the program will be described below.

[0392] User Registration and Login

[0393] A user accesses the system using a terminal and enters the required information (such as name, email address, and password) in a new registration form. The terminal sends this information to the server, which then stores it in a database and creates a new user account. When logging in, the user enters their email address and password from the terminal and sends them to the server, which then references the database to perform authentication and starts a session if authentication is successful.

[0394] Photo upload

[0395] A user uses a device to select multiple photos taken during a trip and request uploading. The device sends the selected photo data to a server, which then stores the received photos in a cloud storage device and records the storage location of each photo in a database.

[0396] Photo analysis

[0397] The server processes the photos stored in the cloud storage device sequentially. Specifically, it uses a generative algorithm (e.g., Google Cloud Vision API) to extract and analyze the photo metadata (date, time, location, etc.). The server stores the analyzed metadata in a database.

[0398] Gathering more information with conversational AI

[0399] Based on the analyzed metadata, the server uses a conversational algorithm (e.g., GPT-3) to generate questions for the user. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. Prompt sentences such as "Where was this photo taken?" or "How did you feel when you took the photo?" may also be used. The device sends the answers to the server, which stores them in a database. The user can also skip questions.

[0400] Emotional regulation by emotion engine

[0401] The emotion recognition engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, when a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion recognition engine analyzes it and generates emotional data. The emotional data is then sent to the server and stored in a database.

[0402] Automatic travel report generation

[0403] The server selects an appropriate travelogue template based on the collected metadata, the user's responses, and sentiment data. For example, it generates a detailed travelogue based on information such as "I took the photo at the top of the Eiffel Tower. The view was amazing." The generated travelogue is then sent to the user's device for viewing.

[0404] Information sharing and virtual travel experiences

[0405] The user sets the travel journal to be public or private using the device. This setting information is sent to the server and stored in a database. The server makes publicly set travel journals available to other users. The server also generates a virtual travel experience based on the published travel journal and provides other users with detailed information on related places and activities.

[0406] In this way, the system of the present invention allows users to efficiently organize their travel photos, automatically generate detailed travel journals, create emotional records, and share them with other users, allowing them to gain information and experiences about new travel destinations.

[0407] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0408] Step 1: User registration and login

[0409] A user accesses the system using a terminal and enters the required information (name, email address, password, etc.) into a new registration form. This becomes input data. The terminal sends this information to the server. The server stores the received information in a database and creates a new user account. This becomes output data. When logging in, the email address and password entered by the user are sent from the terminal to the server. The server refers to the database to perform authentication, and if authentication is successful, a session is started. This allows the user to access the system.

[0410] Step 2: Upload a photo

[0411] A user uses a device to select multiple photos taken during a trip and make an upload request. This is the input data. The device sends the selected photo data to the server. The server stores the received photos in cloud storage and records the storage location of each photo in a database. This is the output data. The photos are then stored in the system.

[0412] Step 3: Photo analysis

[0413] The server sequentially passes the photos stored in the cloud storage device to a generation algorithm (e.g., Google Cloud Vision API). This becomes the input data. The generation algorithm extracts and analyzes the photo's metadata (date, time, location, etc.). The server stores the analyzed metadata in a database. This becomes the output data. As a result, detailed information about the photo is extracted and stored in the database.

[0414] Step 4: Gathering more information with conversational AI

[0415] Based on the analyzed metadata, the server uses an interactive algorithm (e.g., GPT-3) to generate questions for the user. This becomes the input data. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. This becomes the output data. The server sends the generated questions to the user's device, and the user answers the questions. The device sends the answers to the server, and the server stores them in a database. The user can also skip questions, allowing more information to be collected.

[0416] Step 5: Emotional regulation with the Emotion Engine

[0417] The emotion recognition engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. This becomes the input data. For example, when a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion recognition engine analyzes it and generates emotional data. The emotional data is sent to the server and stored in a database. This becomes the output data. This records the user's emotions.

[0418] Step 6: Automatic travel journal generation

[0419] The server selects an appropriate travelogue template based on the collected metadata, user responses, and emotional data. This becomes the input data. A detailed travelogue is generated by incorporating the metadata, episodes, and emotional information into the template. The generated travelogue is then provided to the user's device. This becomes the output data. The user can then view the detailed travelogue.

[0420] Step 7: Information sharing and virtual travel experiences

[0421] The user uses their device to set their travel journal to be public or private. This becomes the input data. The setting information is sent to the server and stored in a database. The server then sets the public travel journal so that other users can view it. This becomes the output data. The server also generates virtual travel experience content based on the published travel journal and provides other users with detailed information about related places and activities. This allows other users to gain information and experiences about new travel destinations.

[0422] (Application example 2)

[0423] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0424] Conventional travel journal creation systems require users to manually organize photos and detailed information to create a travel journal, a time-consuming and labor-intensive process. It is also difficult to reflect the user's emotions, making it difficult to gain emotional empathy when sharing with other users. Furthermore, automatic travel journal generation in video format is limited, making it difficult to provide an effective visual experience.

[0425] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0426] In this invention, the server includes a means for a user to upload multiple images via a device, a means for the server to store the images in cloud storage, a means for a generation algorithm to extract and analyze metadata of the images, a means for an interactive algorithm to ask the user detailed information and store the answers, a means for analyzing the user's emotions in real time using an emotion engine to generate emotion data, a means for automatically generating detailed video recordings based on the generation algorithm, the user's answers, and the emotion data, and a means for enabling the automatically generated recordings to be shared among users. This allows users to easily upload photos, automatically generate emotionally-reflecting travel journals in video format, and share them with other users.

[0427] "User" means a person who uses a digital device to access the system and create and share travel journals.

[0428] "Device" refers to electronic devices with communication capabilities, such as computers, smartphones, and tablets.

[0429] "Images" refers to visual data such as photographs and illustrations taken by users during their travels.

[0430] A "server" is a computer system within a network that receives, processes, and stores data submitted by users.

[0431] "Cloud storage" refers to a service that stores data on remote servers accessible via the Internet.

[0432] A "generative algorithm" is a computational method for automatically generating new content based on input data.

[0433] "Metadata" refers to additional information that accompanies an image, such as the date, time, location, and camera settings.

[0434] An "interactive algorithm" is a program that collects information through dialogue with the user and generates appropriate questions.

[0435] An "emotion engine" is a program that analyzes emotions from user input and actions and generates them as data.

[0436] "Emotion data" is data that indicates the user's emotional state, obtained as a result of analysis by the emotion engine.

[0437] "Video recording" refers to content that integrates images, audio, and text for dynamic playback.

[0438] "Automatically Generated Records" refers to documents or media content that are automatically generated based on data collected by the system.

[0439] "Shareable" means that the records created are viewable by other users or on public platforms, allowing for the widespread dissemination of information.

[0440] A "virtual travel experience" refers to a digital simulation that allows users to experience a place through sight, sound, and other senses without actually visiting the place.

[0441] The present invention provides a system for automatically generating travel video that reflects emotions based on travel photos taken by a user, and sharing the video with other users. Specific embodiments are described below.

[0442] User Registration and Login

[0443] First, a user accesses the system using a device (smartphone or PC) and enters the required information such as name, email address, and password into the new registration form. This information is sent to the server, which then creates a new user account in the database. When logging in, the user enters their email address and password from their device and sends them to the server. The server then refers to the database to perform authentication, and if authentication is successful, a session begins.

[0444] Photo upload

[0445] The user selects multiple photos taken during their trip from their device and makes an upload request. The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[0446] Photo analysis

[0447] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts metadata from the photos and analyzes information such as date, time, and location. The extracted metadata is then stored in a database by the server.

[0448] Gathering more information with conversational AI

[0449] The server uses an interactive algorithm based on the analyzed metadata to generate questions for the user. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. The user can also skip questions. The user's answers are sent from the device to the server, which stores them in a database.

[0450] Emotional regulation by emotion engine

[0451] The emotion engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, if a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion engine analyzes that information and generates emotional data. The emotional data is then sent to a server and stored in a database.

[0452] Automatic travel report generation

[0453] The server selects an appropriate video recording template based on the collected metadata, user responses, and emotion data. Based on the selected template, a detailed travel video is automatically generated. This video recording includes text, photos, anecdotes, and the user's emotions. The generated video recording is then provided to the user's device for viewing.

[0454] Information sharing and virtual travel experiences

[0455] Users can use their devices to set the generated travelogue video as public or private. The setting information is sent to the server and stored in a database. Travelogue videos set as public can be viewed by other users. The server generates a virtual travel experience based on the published travelogue and provides other users with detailed information about related places and activities.

[0456] Specific examples

[0457] For example, suppose a user uploads a photo of Tokyo Tower taken during a trip. The system analyzes the photo's metadata and automatically obtains "Tokyo Tower information." The user then inputs "I had fun at this place" into the emotion engine. Based on this information, a video travelogue is generated as a fun memory.

[0458] Example prompts for generative AI models

[0459] Upload a photo of Tokyo Tower and write a travel journal about the fun you had.

[0460] In this way, the system of the present invention allows users to easily organize travel photos, automatically create travel journals that reflect their emotions, and share them with other users.

[0461] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0462] Step 1: User registration and login

[0463] First, a user accesses the system using a terminal and enters the required information such as name, email address, and password into a new registration form. The terminal sends this information to the server, which receives the information and creates a new user account in the database. When logging in, the user enters their email address and password from the terminal and sends them to the server. The server refers to the database to perform authentication, and if authentication is successful, a session begins.

[0464] Input: User information (name, email address, password)

[0465] Output: User account creation and authentication session

[0466] Step 2: Upload a photo

[0467] The user selects multiple photos taken during their trip from their device and makes an upload request. The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[0468] Input: A photo selected by the user

[0469] Output: Photos saved in cloud storage and a record of their location

[0470] Step 3: Photo analysis

[0471] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts metadata from the photos and analyzes information such as date, time, and location. The extracted metadata is then stored in a database by the server.

[0472] Input: Photos stored in cloud storage

[0473] Output: Extracted and parsed metadata (date, time, location, etc.)

[0474] Step 4: Gathering more information with conversational AI

[0475] The server uses an interactive algorithm based on the analyzed metadata to generate questions for the user. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. The user can also skip questions. The user's answers are sent from the device to the server, which stores them in a database.

[0476] Input: Parsed metadata

[0477] Output: User details and answer questions

[0478] Step 5: Emotional regulation with the Emotion Engine

[0479] The emotion engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, if a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion engine analyzes that information and generates emotional data. The emotional data is then sent to a server and stored in a database.

[0480] Input: User's emotional expression

[0481] Output: Generated emotion data

[0482] Step 6: Automatic travel journal generation

[0483] The server selects an appropriate video recording template based on the collected metadata, user responses, and emotion data. Based on the selected template, a detailed travel video is automatically generated. This video recording includes text, photos, anecdotes, and the user's emotions. The generated video recording is then provided to the user's device.

[0484] Input: Metadata, user responses, sentiment data

[0485] Output: Automatically generated travel video

[0486] Step 7: Information sharing and virtual travel experiences

[0487] Users can use their devices to set the generated travelogue video as public or private. The setting information is sent to the server and stored in a database. Travelogue videos set as public can be viewed by other users. The server generates a virtual travel experience based on the published travelogue and provides other users with detailed information about related places and activities.

[0488] Input: Information set to public or private

[0489] Output: Published travelogues and virtual travel experiences

[0490] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0491] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0492] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0493] [Second embodiment]

[0494] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0495] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0496] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0497] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0498] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0499] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0500] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0501] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0502] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0503] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0504] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0505] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0506] This invention relates to a system that allows a user to upload multiple photos taken during a trip, analyzes the date, time, location, and detailed information from those photos, and automatically generates a travel journal. Specific embodiments for implementing this system and their program processing are described below.

[0507] User Registration and Login

[0508] A user accesses the system using a terminal and first registers, entering information such as name, email address, and password into the registration form, which the terminal then sends to the server, which receives the information and creates a new user account in the database.

[0509] When logging in, the user enters their email address and password from their device and sends them to the server. The server then refers to the database for authentication, and if successful, starts a session.

[0510] Photo upload

[0511] The user uploads multiple photos taken during their trip from their device. The device sends a request to upload the selected photos to the server, and the server stores the received photos in cloud storage. The server records the storage location of each photo in a database.

[0512] Photo analysis

[0513] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts and analyzes the photo's metadata (such as the date and time of the photo's capture, GPS information, etc.). The extracted metadata is then stored in a database by the server.

[0514] Gathering more information with conversational AI

[0515] The server uses an interactive algorithm to generate questions based on the parsed metadata. For example, questions like "Where was this photo taken?" or "What experience did you have here?" are sent to the user's device, and the user answers them. The user can also skip questions. The device sends the user's answers to the server, which stores them in a database.

[0516] Automatic travel report generation

[0517] The server selects an appropriate travelogue template based on the collected information (analysis metadata and user responses) and automatically generates a detailed travelogue, which includes text, photos, anecdotes, etc. The generated travelogue is then delivered to the user's device for viewing.

[0518] Information sharing and virtual travel experiences

[0519] Users can set their travel journals to be public or private using their devices. The setting information is sent to the server, which stores it in a database. Travel journals set to public can be viewed by other users. The server generates a virtual travel experience based on the published travel journal and provides it to other users, along with detailed information about related places and activities.

[0520] These processes allow users to easily organize their travel photos, automatically generate detailed and engaging travel journals, and share them with other users to enjoy virtual travel experiences.

[0521] The processing flow will be explained below.

[0522] Step 1:

[0523] A user accesses the system using a terminal and enters the required information, such as name, email address, and password, into the new registration form.

[0524] Step 2:

[0525] The device sends the user's input information to the server and makes a registration request.

[0526] Step 3:

[0527] The server takes the information you enter and creates a new user account in its database.

[0528] Step 4:

[0529] From next time onwards, the user will enter their email address and password into the login form on their device to request a login.

[0530] Step 5:

[0531] The terminal sends authentication information to the server, and the server performs authentication by referencing a database.

[0532] Step 6:

[0533] If the server successfully authenticates the user, it starts a session for the user and grants access rights.

[0534] Step 7:

[0535] The user uses the device to select multiple photos taken during the trip and submits an upload request.

[0536] Step 8:

[0537] The device transmits the selected photo data to the server, and the server stores the photos in cloud storage.

[0538] Step 9:

[0539] The server records the location of each photo in a database.

[0540] Step 10:

[0541] The server sequentially passes the photos stored in cloud storage to the generation algorithm.

[0542] Step 11:

[0543] The generation algorithm extracts and analyzes the photo's metadata (such as the date and time the photo was taken and GPS information).

[0544] Step 12:

[0545] The server stores the extracted metadata in a database and organizes the parsed information.

[0546] Step 13:

[0547] The server uses an interactive algorithm to generate questions for the user based on the parsed metadata.

[0548] Step 14:

[0549] The interactive algorithm generates questions and sends them to the user's device.

[0550] Step 15:

[0551] The user uses the device to answer questions posed by the conversational algorithm or to skip questions.

[0552] Step 16:

[0553] The device sends the user's answers to the server, which stores them in a database.

[0554] Step 17:

[0555] The server selects an appropriate travelogue template based on the information it collects (metadata and user responses).

[0556] Step 18:

[0557] The server fills in the information in the selected template and automatically generates a detailed travelogue.

[0558] Step 19:

[0559] The server provides the generated travelogue to the user's terminal, and the user views the travelogue.

[0560] Step 20:

[0561] Users can use their devices to set their travel journals to be public or private.

[0562] Step 21:

[0563] The terminal sends the configuration information to the server, which stores it in a database.

[0564] Step 22:

[0565] Other users can use their devices to view your published travelogue.

[0566] Step 23:

[0567] The server generates a virtual travel experience based on the published travelogue and provides it to other users.

[0568] Step 24:

[0569] The server also provides detailed information about related places and activities to support virtual travel.

[0570] These steps allow users to easily organize their travel photos, automatically generate and share detailed travel journals, and share their travel journals with other users to enjoy virtual travel experiences.

[0571] Example 1

[0572] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0573] Efficiently organizing the many images taken during a trip and generating a detailed travelogue based on them is a time-consuming and labor-intensive process. There is also a need for a method to share travelogues with other users and provide them with virtual travel experiences. However, existing systems lack the functionality to allow users to easily analyze image metadata and automatically generate detailed travelogues. Furthermore, they lack an interactive algorithm that generates questions based on user answers, and an effective means to provide virtual travel experiences.

[0574] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0575] In this invention, the server includes a means for users to upload multiple images via their terminals, a means for the server to store the images in cloud storage, a means for a generation algorithm to extract and analyze metadata from the images, a means for an interactive algorithm to ask the user detailed information and store the answers, a means for automatically generating a detailed travelogue based on the generation algorithm and the user's answers, and a means for enabling the automatically generated travelogue to be shared among users. This allows users to easily upload images taken during their trip and automatically generate and share a detailed travelogue based on the images. A virtual travel experience is also provided, allowing other users to share their travelogues and enjoy virtual experiences.

[0576] "User" means the person who uses the system to upload images or input information.

[0577] A "terminal" is an electronic device such as a computer or smartphone used by a user.

[0578] "Images" are photo files taken by users during their travels and uploaded to the system.

[0579] The "server" refers to a central management system that stores images and analyzes metadata.

[0580] "Cloud storage" is an external service for storing and managing data via the Internet.

[0581] "Generation algorithm" refers to software that extracts and analyzes image metadata.

[0582] "Metadata" is additional information including the date and time an image was taken and GPS information.

[0583] An "interactive algorithm" is software that generates questions to gather more information from users and stores their answers.

[0584] A "travel journal" is a written record generated based on images and information provided by the user.

[0585] "Virtual Travel Experience" is a feature that allows other users to virtually experience travel based on automatically generated travelogues.

[0586] This invention relates to a system that allows a user to upload multiple images taken during a trip, analyzes the date, time, location, and detailed information from those images, and automatically generates a travelogue. Specific embodiments for implementing this system are described below.

[0587] User Registration and Login

[0588] A user accesses the system using a terminal and first registers. The user enters their name, email address, and password into a registration form, and the terminal sends this information to the server. The server receives the entered information and creates a new user account in the database. When the user logs in, they enter their email address and password and send them to the server. The server refers to the database to authenticate them, and if successful, starts a session.

[0589] Specific behavior:

[0590] The user fills in the required information in the registration form and clicks the "Register" button.

[0591] The device sends the input to the server.

[0592] The server creates a new user account in the database.

[0593] Photo upload

[0594] The user uploads multiple photos taken during their trip from their device. The device sends a request to upload the selected photos to the server, and the server stores the received photos in cloud storage such as Google Cloud Storage. The server records the storage location of each photo in a database.

[0595] Specific behavior:

[0596] The user selects multiple photos from a file folder and clicks the "Upload" button.

[0597] The terminal transmits the selected photo file to the server.

[0598] The server stores the photo in Google Cloud Storage and records the URL in a database.

[0599] Photo analysis

[0600] The server sequentially retrieves photos stored in cloud storage and extracts the photo metadata (date and time of shooting, GPS information, etc.) using image analysis libraries such as OpenCV and TensorFlow. The extracted metadata is then stored in a database by the server.

[0601] Specific behavior:

[0602] The server retrieves the photos from the cloud storage.

[0603] The server uses an image analysis library to extract the metadata.

[0604] The extracted metadata is stored in a database.

[0605] Gathering more information with conversational AI

[0606] The server uses an interactive algorithm to generate questions based on the parsed metadata, sends the generated questions to the user's device, and the user can answer or skip the questions. The device then sends the user's answers to the server, which stores them in a database.

[0607] Specific behavior:

[0608] The server generates questions based on the metadata.

[0609] Example questions: "Where was this photo taken?"

[0610] The device displays a question to the user.

[0611] The user enters the answer and the device sends the answer to the server.

[0612] Automatic travel report generation

[0613] The server selects an appropriate travelogue template based on the collected information (analysis metadata and user responses), and automatically generates a detailed travelogue using a generative AI model such as GPT-3. The generated travelogue is then sent to the user's device for viewing.

[0614] Example prompt sentence:

[0615] "These are travelogues to Tokyo. Please generate a detailed travelogue based on the information below."

[0616] Specific behavior:

[0617] The server sends a prompt to the generative AI model.

[0618] The generated travelogue is sent to the user's device.

[0619] Information sharing and virtual travel experiences

[0620] Users set their travel journals to be public or private and send the setting information to the server. Travel journals that are public can be viewed by other users. The server generates a virtual travel experience based on the published travel journal and provides it to other users.

[0621] Specific behavior:

[0622] Users can set their travel journals to be public or private.

[0623] The server acquires publicly set travelogues and generates a virtual travel experience.

[0624] By implementing this system, users can easily organize their travel photos, automatically generate detailed and attractive travel journals, and share them with other users to enjoy virtual travel experiences.

[0625] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0626] Step 1: User Registration

[0627] The user enters their name, email address, and password in plain text.

[0628] The device sends this input information to the server in JSON format.

[0629] The server stores the received information in its database as a new user account.

[0630] Enter your name, email address, and password.

[0631] Data processing: Convert input information into JSON format.

[0632] Output: A new user account is created in the database.

[0633] Specific behavior:

[0634] The user enters "Yamada Taro", "taro@example.com", and a password in the registration form and clicks the "Register" button.

[0635] The device sends this information to the server in JSON format.

[0636] The server receives the information and creates a new user account in the database.

[0637] Step 2: Log in

[0638] The user enters their email address and password in plain text.

[0639] The device sends this input information to the server in JSON format.

[0640] The server refers to the database and performs authentication. If authentication is successful, a session is started.

[0641] Enter your email address and password.

[0642] Data calculation: Matching input information with existing information in a database.

[0643] Output: A session ID is generated.

[0644] Specific behavior:

[0645] The user enters "taro@example.com" and their password in the login form and clicks the "Login" button.

[0646] The device sends this information to the server in JSON format.

[0647] The server queries the database, and since authentication is successful, generates a session ID.

[0648] Step 3: Upload your photos

[0649] The user selects multiple photos taken during their trip from their device. The selection is made in file format.

[0650] The device sends the selected photo file to the server via an HTTP POST request.

[0651] The server stores the received photos in Google Cloud Storage and records their URLs in the database.

[0652] Input: Travel photo files.

[0653] Data processing: Convert the photo file into HTTP POST request format.

[0654] Output: The photo is saved to Google Cloud Storage and its URL is recorded in the database.

[0655] Specific behavior:

[0656] The user selects "IMG_001.jpg" to "IMG_010.jpg" from the file folder and clicks the "Upload" button.

[0657] The device sends the selected photo file to the server via an HTTP POST request.

[0658] The server stores the received photos in Google Cloud Storage and records their URLs in the database.

[0659] Step 4: Photo analysis

[0660] The server retrieves photos sequentially from Google Cloud Storage.

[0661] The server uses image analysis libraries such as OpenCV and TensorFlow to extract photo metadata (such as the date and time the photo was taken, GPS information, etc.).

[0662] The server stores the extracted metadata in a database.

[0663] Input: Google Cloud Storage photo URL.

[0664] Data Computation: Extraction of metadata using image analysis libraries.

[0665] Output: The extracted metadata is stored in a database.

[0666] Specific behavior:

[0667] The server retrieves "IMG_001.jpg" from Google Cloud Storage.

[0668] The server analyzes the EXIF ​​information of the photo using OpenCV and extracts the date and time of the photo and GPS information.

[0669] The extracted metadata is stored in a database.

[0670] Step 5: Gathering more information with conversational AI

[0671] The server generates questions based on the parsed metadata, using a generative AI model (e.g., GPT-3).

[0672] The device displays the generated question to the user, who can then enter an answer or skip.

[0673] The device sends the user's answers to the server, which stores them in a database.

[0674] Input: Metadata.

[0675] Data Computing: Generate questions and store answers using generative AI models.

[0676] Output: The generated questions and the user's answers are stored in a database.

[0677] Specific behavior:

[0678] Based on the metadata, the server sends the question "Where was this photo taken?" as a prompt to the generative AI model and receives the question as an answer.

[0679] The device displays a question to the user.

[0680] The user answers "Tokyo Tower," and the device sends the answer to the server.

[0681] The server stores the answer in a database.

[0682] Step 6: Automatic travel journal generation

[0683] The server selects an appropriate travelogue template based on the collected metadata and the user's answers.

[0684] The server automatically generates a detailed travelogue using a generative AI model (e.g., GPT-3).

[0685] The server sends the generated travelogue to the user's terminal.

[0686] Input: Metadata, user answers.

[0687] Data computation: Generating travel journals using generative AI models.

[0688] Output: The automatically generated travelogue is sent to the user's device.

[0689] Specific behavior:

[0690] The server converts the metadata and answers into prompts and sends them to the generative AI model.

[0691] The generated travelogue is sent to the user's device and can be viewed by the user.

[0692] Step 7: Information sharing and virtual travel experiences

[0693] The user sets the travel journal to be public or private and sends it from the device to the server.

[0694] The server stores the configuration information in a database.

[0695] The server generates a virtual travel experience based on the published travelogue and provides it to other users.

[0696] Input: Public / Private settings.

[0697] Data calculation: Converting published travel journal data into virtual experience content.

[0698] Output: The publicly set travelogue is shared with other users, providing them with a virtual travel experience.

[0699] Specific behavior:

[0700] The user sets whether their travel journal is public or private, and sends the setting information from their device to the server.

[0701] The server stores the configuration information in a database.

[0702] The server acquires the published travelogue data, generates a virtual travel experience, and provides it to other users.

[0703] The above is the specific processing flow of this system and the operation at each step.

[0704] (Application example 1)

[0705] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0706] In today's world, many users find it time-consuming to simply organize photos taken during a trip and record their experiences in a detailed travel journal. It's also difficult to share those travel journals with others or provide virtual travel experiences. This often results in valuable travel experiences remaining buried. The present invention aims to solve these problems by providing a system that automatically and efficiently generates travel journals based on photos taken by users and allows them to publish and share them in a virtual store.

[0707] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0708] In this invention, the server includes means for a user to upload multiple images via a terminal, means for the server to store the images in cloud storage, means for a generation algorithm to extract and analyze metadata from the images, means for an interactive algorithm to ask the user detailed information and store the answers, means for automatically generating detailed records based on the generation algorithm and the user's answers, means for enabling the automatically generated records to be shared among users, means for providing a virtual experience based on the collected information and the generated records, and means for publishing the generated records in a virtual store. This allows users to efficiently organize travel photos, automatically generate and publish attractive travel journals, and share and enjoy them with other users.

[0709] "User" means any person or entity that accesses the system, uploads multiple images, and inputs information to generate detailed records from those images.

[0710] "Terminal" means the hardware device used by a User to access the System and upload images or enter information.

[0711] "Server" refers to the back-end computing resources used to store images received from users and run generative and interactive algorithms.

[0712] "Cloud storage" refers to an external storage system that allows a server to safely store image data received via the Internet.

[0713] "Generation Algorithm" refers to the software process used to extract and analyze metadata from uploaded images and automatically generate detailed records.

[0714] "Metadata" refers to data that contains additional information specific to an image, such as the date and time the image was taken or GPS information.

[0715] "Interactive algorithm" refers to the software process by which a system generates questions to gather more information from users and stores their answers.

[0716] "Detailed Records" refers to travel journals and other documents that are automatically generated based on images and information provided by users.

[0717] "Virtual Store" refers to an online platform for publishing and sharing detailed user-generated records with other users.

[0718] A "virtual experience" is an experience that allows other users to take a virtual journey based on a publicly detailed record.

[0719] This invention relates to a system that allows users to upload multiple images taken during their trip, analyzes the date, time, location, and detailed information from those images, and automatically generates a detailed record. Specific embodiments for implementing this system and their program processing are described below.

[0720] User Registration and Login

[0721] A user accesses the system using a terminal and first registers. They enter information such as their name, email address, and password into the registration form, and the terminal sends this information to the server. The server receives the entered information and creates a new user account in its database. When logging in, the user enters their email address and password from the terminal and sends them to the server. The server refers to the database to perform authentication, and if successful, starts a session.

[0722] Photo upload

[0723] The user uploads multiple photos taken during their trip from their device. The device sends a request to upload the selected photos to the server, which then stores the photos in cloud storage. The server records the location of each photo in a database.

[0724] Photo analysis

[0725] The server sequentially passes photos stored in the cloud storage to a generation algorithm, which extracts and analyzes the photo's metadata (such as the date and time of the photo's capture, GPS information, etc.) The extracted metadata is then stored in a database by the server.

[0726] Gathering more information with conversational AI

[0727] The server uses an interactive algorithm to generate questions based on the parsed metadata. For example, questions like "Where was this photo taken?" or "What experience did you have here?" are sent to the user's device, and the user answers them. The user can also skip questions. The device sends the user's answers to the server, which stores them in a database.

[0728] Examples of specific prompts include:

[0729] 1. Question: "Where was this photo taken?"

[0730] User Answer: "This photo was taken at Kiyomizu-dera Temple in Kyoto."

[0731] 2. Question: "What was your experience here?"

[0732] User Answer: "I enjoyed visiting tourist spots and eating my way around."

[0733] Detailed automatic recording

[0734] The server selects an appropriate template based on the collected information (analysis metadata and user responses) and automatically generates a detailed record, which includes text, photos, anecdotes, etc. The generated record is then delivered to the user's device for viewing.

[0735] Virtual Experience and Information Sharing

[0736] Users can use their devices to set their detailed records to public or private. The setting information is sent to the server, which stores it in a database. Publicly set detailed records can be viewed by other users. The server generates a virtual experience based on the publicly set detailed records and provides it to other users, along with detailed information about related locations and activities.

[0737] The system is implemented using the following hardware and software:

[0738] Server: Cloud server, database (e.g., Amazon RDS)

[0739] Image storage: Cloud storage (e.g. AWS S3)

[0740] Generative AI models: Libraries for running generative algorithms, such as TensorFlow and PyTorch

[0741] This allows users to efficiently organize their travel photos, automatically generate and publish compelling travel journals, and share and enjoy them with other users.

[0742] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0743] Step 1: User registration and login

[0744] A user accesses the system using a terminal, enters their name, email address, and password, and submits it. The server receives this information and creates a new user account in the database. When logging in, the user enters their email address and password. The server uses this information to look up their database and authenticates them, and if successful, starts a session.

[0745] Input: User's name, email address, password

[0746] Output: New user account created and authenticated.

[0747] Step 2: Upload a photo

[0748] Users select multiple travel photos from their devices and send an upload request to the server, which then stores the photos in cloud storage and records their location in a database.

[0749] Input: Multiple photos taken by the user

[0750] Output: Save photos to cloud storage and record their location in a database

[0751] Step 3: Photo analysis

[0752] The server passes the photos stored in the cloud storage to a generation algorithm, which extracts and analyzes the photo's metadata (date and time of the photo, GPS information), which is then stored in a database by the server.

[0753] Input: Photos stored in cloud storage

[0754] Output: Extracted metadata stored in a database

[0755] Step 4: Gathering more information with conversational AI

[0756] The server uses an interactive algorithm to generate questions based on the parsed metadata. The generated questions are sent to the user's device, and the user answers them. The device then sends the user's answers to the server, which stores them in a database.

[0757] Input: extracted metadata, user answers

[0758] Output: Generated questions, saving user answers to database

[0759] Examples of specific prompts include:

[0760] 1. Question: "Where was this photo taken?"

[0761] User Answer: "This photo was taken at Kiyomizu-dera Temple in Kyoto."

[0762] 2. Question: "What was your experience here?"

[0763] User Answer: "I enjoyed visiting tourist spots and eating my way around."

[0764] Step 5: Automatically generate detailed records

[0765] The server automatically generates a detailed record based on the generation algorithm and the user's answers. The generated record includes text, photos, anecdotes, etc. The server then provides the generated record to the user's device.

[0766] Input: Analysis metadata, user answers

[0767] Output: Auto-generated detailed transcript

[0768] Step 6: Virtual Experience and Information Sharing

[0769] Users can use their devices to set their detailed records to public or private. The server stores this setting information in a database. Publicly set detailed records can be viewed by other users. The server generates a virtual experience based on the publicly set detailed records and provides it to other users.

[0770] Input: User's public / private settings

[0771] Output: Publishing detailed records and generating virtual experiences

[0772] These processing flows allow users to efficiently organize photos taken during their trip, automatically generate and publish detailed and engaging travel journals, and share their virtual experiences with other users.

[0773] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0774] The present invention relates to a system that allows users to upload multiple photos taken during their trip, analyzes the date, time, location, and detailed information from those photos, and automatically generates a travel diary. Furthermore, by combining this system with an emotion engine, the system can recognize the user's emotions and reflect them in the travel diary. Specific embodiments of the present invention and the processing of the program are described below.

[0775] User Registration and Login

[0776] A user accesses the system using a terminal and enters the required information such as name, email address, password, etc. into a new registration form. The terminal sends this information to the server, which receives it and creates a new user account in the database.

[0777] When logging in, the user enters their email address and password from their device and sends them to the server. The server then refers to the database for authentication, and if authentication is successful, a session begins.

[0778] Photo upload

[0779] The user selects multiple photos taken during their trip using their device and makes an upload request. The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[0780] Photo analysis

[0781] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts metadata from the photos and analyzes information such as date, time, and location. The extracted metadata is then stored in a database by the server.

[0782] Gathering more information with conversational AI

[0783] The server uses an interactive algorithm to generate questions for the user based on the parsed metadata. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. The user can also skip questions. The device sends the answers to the server, which stores them in a database.

[0784] Emotional regulation by emotion engine

[0785] The emotion engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, when a user inputs an emotion expression such as "I had fun" or "I'm a little tired," the emotion engine analyzes the information and generates emotion data. The emotion data is then sent to the server and stored in a database.

[0786] Automatic travel report generation

[0787] The server selects an appropriate travelogue template based on the collected metadata, user responses, and emotion data. A detailed travelogue is automatically generated based on the selected template. This travelogue includes text, photos, anecdotes, and the user's emotions. The generated travelogue is then provided to the user's device for viewing.

[0788] Information sharing and virtual travel experiences

[0789] Users can set their travel journals to be public or private using their devices. The setting information is sent to the server and stored in a database. Travel journals set to public can be viewed by other users. The server generates a virtual travel experience based on the published travel journal and provides other users with detailed information on related places and activities.

[0790] For example, if a user uploads photos taken during a trip and inputs the emotion "It was fun" through the emotion engine, the system will analyze this information and reflect it in the travelogue as a "fun memory." In addition, when other users view this travelogue, content will be provided that allows them to feel a "fun atmosphere" as if they are traveling virtually.

[0791] This allows the system of the present invention to not only allow users to easily organize their travel photos and automatically generate detailed travel journals, but also to create records that reflect the user's emotions and share them with other users. Furthermore, other users can gain information and experiences about new travel destinations through virtual travel experiences.

[0792] The processing flow will be explained below.

[0793] Step 1:

[0794] A user accesses the system using a terminal and enters the required information, such as name, email address, and password, into the new registration form.

[0795] Step 2:

[0796] The terminal sends the entered user information to the server, which then creates a new user account in the database.

[0797] Step 3:

[0798] The user uses a device to enter their email address and password into the login form to request a login.

[0799] Step 4:

[0800] The terminal sends authentication information to the server, which then refers to the database to authenticate the user. If authentication is successful, the server starts a session for the user.

[0801] Step 5:

[0802] A user uses a device to select multiple travel photos and make an upload request.

[0803] Step 6:

[0804] The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[0805] Step 7:

[0806] The server sequentially passes the photos stored in cloud storage to the generation algorithm.

[0807] Step 8:

[0808] The generation algorithm extracts and analyzes the metadata of the photo (such as the date and time of the photo being taken, GPS information, etc.), and the extracted metadata is stored in a database by the server.

[0809] Step 9:

[0810] The server uses an interactive algorithm to generate questions for the user based on the parsed metadata.

[0811] Step 10:

[0812] The conversational algorithm generates questions and sends them to the user's device, such as "Where was this photo taken?" or "What kind of experience did you have here?"

[0813] Step 11:

[0814] The user uses the device to answer questions posed by the conversational algorithm or to skip questions.

[0815] Step 12:

[0816] The device sends the user's answers to the server, which stores them in a database.

[0817] Step 13:

[0818] The conversational algorithm uses an emotion engine to recognize the user's emotions. For example, if the user answers "I had fun" or "I'm a little tired," the emotion will be recognized.

[0819] Step 14:

[0820] The emotion engine generates the recognized emotion data and sends it to the server, which stores the emotion data in a database.

[0821] Step 15:

[0822] The server selects an appropriate travel diary template based on the collected metadata, user responses, and sentiment data.

[0823] Step 16:

[0824] The server automatically generates a detailed travelogue by filling in the selected template with information, including text, photos, anecdotes, and the user's emotions.

[0825] Step 17:

[0826] The server provides the generated travelogue to the user's terminal, and the user views the travelogue.

[0827] Step 18:

[0828] Users can use their devices to set their travel journals to be public or private.

[0829] Step 19:

[0830] The terminal sends the configuration information to the server, which stores the configuration information in a database.

[0831] Step 20:

[0832] Other users can use their devices to view your published travelogue.

[0833] Step 21:

[0834] The server generates a virtual travel experience based on the published travelogue. Emotional data is also taken into consideration, and content adapted to the user's emotions is provided in the virtual travel experience.

[0835] Step 22:

[0836] The server provides detailed information about places and activities related to the virtual travel experience, and helps other users gain information and experiences about new travel destinations through the travelogue.

[0837] These steps allow users to easily organize their travel photos, automatically generate and share detailed and emotional travel journals, and allow other users to experience virtual travel through the journals.

[0838] Example 2

[0839] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0840] Conventional methods require users to organize multiple photos taken during a trip and automatically generate a travelogue by analyzing the date, time, location, and detailed information from those photos. This requires a lot of effort and effort. It is also difficult to create a travelogue that reflects emotions, and it is not easy to share or provide a virtual travel experience. There is a need for a system that solves these problems and enables efficient automatic generation and sharing of travelogues that reflect emotions.

[0841] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0842] In this invention, the server includes means for a user to upload multiple images via a device, means for the server to store the images in a cloud storage device, means for a generation algorithm to extract and analyze metadata for the images, means for an interactive algorithm to ask the user detailed information and store the responses, means for an emotion recognition engine to recognize the user's emotions and analyze and store the emotion data, means for automatically generating detailed travelogues based on the generation algorithm, the user's responses, and the emotion data, and means for enabling the automatically generated travelogues to be shared among users, thereby enabling users to efficiently organize photos taken during their trips, automatically generate detailed travelogues, and create travelogues that reflect their emotions and share them with other users.

[0843] "User" refers to any individual or entity that uses the System to generate travel journals and upload photos.

[0844] "Device" refers to the hardware used by a user, such as a computer, smartphone, or tablet.

[0845] "Images" refers to photographic data taken by users during their travels and uploaded to the system.

[0846] "Server" refers to a computer system that controls the entire system and exchanges data with users.

[0847] "Cloud storage" refers to a storage service for storing data via the Internet.

[0848] "Generation algorithm" refers to a program for extracting and analyzing image metadata.

[0849] "Metadata" refers to additional information contained in an image, such as the date, time, location, and shooting conditions.

[0850] An "interactive algorithm" is a program designed to ask users detailed questions and collect their answers.

[0851] "Emotion recognition engine" refers to a system for analyzing and storing user emotional data.

[0852] "Emotional data" refers to information analyzed by an emotion recognition engine based on emotional expressions entered by the user.

[0853] "Records" refers to detailed travel journals that are automatically generated based on generative algorithms, user responses, and sentiment data.

[0854] "Virtual travel experience" refers to content that allows other users to experience the atmosphere and information of a trip based on publicly available records.

[0855] "Publicity Settings" refers to the settings options that allow a User to make a Recording that the User generates available to other Users.

[0856] The present invention relates to a system that allows users to upload multiple photos taken during their trip, analyzes the date, time, location, and detailed information from those photos, and automatically generates a travel diary. Furthermore, by combining this system with an emotion recognition engine, the system can recognize the user's emotions and reflect them in the travel diary. Specific embodiments of the present invention and the processing of the program will be described below.

[0857] User Registration and Login

[0858] A user accesses the system using a terminal and enters the required information (such as name, email address, and password) in a new registration form. The terminal sends this information to the server, which then stores it in a database and creates a new user account. When logging in, the user enters their email address and password from the terminal and sends them to the server, which then references the database to perform authentication and starts a session if authentication is successful.

[0859] Photo upload

[0860] A user uses a device to select multiple photos taken during a trip and request uploading. The device sends the selected photo data to a server, which then stores the received photos in a cloud storage device and records the storage location of each photo in a database.

[0861] Photo analysis

[0862] The server processes the photos stored in the cloud storage device sequentially. Specifically, it uses a generative algorithm (e.g., Google Cloud Vision API) to extract and analyze the photo metadata (date, time, location, etc.). The server stores the analyzed metadata in a database.

[0863] Gathering more information with conversational AI

[0864] Based on the analyzed metadata, the server uses a conversational algorithm (e.g., GPT-3) to generate questions for the user. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. Prompt sentences such as "Where was this photo taken?" or "How did you feel when you took the photo?" may also be used. The device sends the answers to the server, which stores them in a database. The user can also skip questions.

[0865] Emotional regulation by emotion engine

[0866] The emotion recognition engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, when a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion recognition engine analyzes it and generates emotional data. The emotional data is then sent to the server and stored in a database.

[0867] Automatic travel report generation

[0868] The server selects an appropriate travelogue template based on the collected metadata, the user's responses, and sentiment data. For example, it generates a detailed travelogue based on information such as "I took the photo at the top of the Eiffel Tower. The view was amazing." The generated travelogue is then sent to the user's device for viewing.

[0869] Information sharing and virtual travel experiences

[0870] The user sets the travel journal to be public or private using the device. This setting information is sent to the server and stored in a database. The server makes publicly set travel journals available to other users. The server also generates a virtual travel experience based on the published travel journal and provides other users with detailed information on related places and activities.

[0871] In this way, the system of the present invention allows users to efficiently organize their travel photos, automatically generate detailed travel journals, create emotional records, and share them with other users, allowing them to gain information and experiences about new travel destinations.

[0872] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0873] Step 1: User registration and login

[0874] A user accesses the system using a terminal and enters the required information (name, email address, password, etc.) into a new registration form. This becomes input data. The terminal sends this information to the server. The server stores the received information in a database and creates a new user account. This becomes output data. When logging in, the email address and password entered by the user are sent from the terminal to the server. The server refers to the database to perform authentication, and if authentication is successful, a session is started. This allows the user to access the system.

[0875] Step 2: Upload a photo

[0876] A user uses a device to select multiple photos taken during a trip and make an upload request. This is the input data. The device sends the selected photo data to the server. The server stores the received photos in cloud storage and records the storage location of each photo in a database. This is the output data. The photos are then stored in the system.

[0877] Step 3: Photo analysis

[0878] The server sequentially passes the photos stored in the cloud storage device to a generation algorithm (e.g., Google Cloud Vision API). This becomes the input data. The generation algorithm extracts and analyzes the photo's metadata (date, time, location, etc.). The server stores the analyzed metadata in a database. This becomes the output data. As a result, detailed information about the photo is extracted and stored in the database.

[0879] Step 4: Gathering more information with conversational AI

[0880] Based on the analyzed metadata, the server uses an interactive algorithm (e.g., GPT-3) to generate questions for the user. This becomes the input data. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. This becomes the output data. The server sends the generated questions to the user's device, and the user answers the questions. The device sends the answers to the server, and the server stores them in a database. The user can also skip questions, allowing more information to be collected.

[0881] Step 5: Emotional regulation with the Emotion Engine

[0882] The emotion recognition engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. This becomes the input data. For example, when a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion recognition engine analyzes it and generates emotional data. The emotional data is sent to the server and stored in a database. This becomes the output data. This records the user's emotions.

[0883] Step 6: Automatic travel journal generation

[0884] The server selects an appropriate travelogue template based on the collected metadata, user responses, and emotional data. This becomes the input data. A detailed travelogue is generated by incorporating the metadata, episodes, and emotional information into the template. The generated travelogue is then provided to the user's device. This becomes the output data. The user can then view the detailed travelogue.

[0885] Step 7: Information sharing and virtual travel experiences

[0886] The user uses their device to set their travel journal to be public or private. This becomes the input data. The setting information is sent to the server and stored in a database. The server then sets the public travel journal so that other users can view it. This becomes the output data. The server also generates virtual travel experience content based on the published travel journal and provides other users with detailed information about related places and activities. This allows other users to gain information and experiences about new travel destinations.

[0887] (Application example 2)

[0888] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0889] Conventional travel journal creation systems require users to manually organize photos and detailed information to create a travel journal, a time-consuming and labor-intensive process. It is also difficult to reflect the user's emotions, making it difficult to gain emotional empathy when sharing with other users. Furthermore, automatic travel journal generation in video format is limited, making it difficult to provide an effective visual experience.

[0890] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0891] In this invention, the server includes a means for a user to upload multiple images via a device, a means for the server to store the images in cloud storage, a means for a generation algorithm to extract and analyze metadata of the images, a means for an interactive algorithm to ask the user detailed information and store the answers, a means for analyzing the user's emotions in real time using an emotion engine to generate emotion data, a means for automatically generating detailed video recordings based on the generation algorithm, the user's answers, and the emotion data, and a means for enabling the automatically generated recordings to be shared among users. This allows users to easily upload photos, automatically generate emotionally-reflecting travel journals in video format, and share them with other users.

[0892] "User" means a person who uses a digital device to access the system and create and share travel journals.

[0893] "Device" refers to electronic devices with communication capabilities, such as computers, smartphones, and tablets.

[0894] "Images" refers to visual data such as photographs and illustrations taken by users during their travels.

[0895] A "server" is a computer system within a network that receives, processes, and stores data submitted by users.

[0896] "Cloud storage" refers to a service that stores data on remote servers accessible via the Internet.

[0897] A "generative algorithm" is a computational method for automatically generating new content based on input data.

[0898] "Metadata" refers to additional information that accompanies an image, such as the date, time, location, and camera settings.

[0899] An "interactive algorithm" is a program that collects information through dialogue with the user and generates appropriate questions.

[0900] An "emotion engine" is a program that analyzes emotions from user input and actions and generates them as data.

[0901] "Emotion data" is data that indicates the user's emotional state, obtained as a result of analysis by the emotion engine.

[0902] "Video recording" refers to content that integrates images, audio, and text for dynamic playback.

[0903] "Automatically Generated Records" refers to documents or media content that are automatically generated based on data collected by the system.

[0904] "Shareable" means that the records created are viewable by other users or on public platforms, allowing for the widespread dissemination of information.

[0905] A "virtual travel experience" refers to a digital simulation that allows users to experience a place through sight, sound, and other senses without actually visiting the place.

[0906] The present invention provides a system for automatically generating travel video that reflects emotions based on travel photos taken by a user, and sharing the video with other users. Specific embodiments are described below.

[0907] User Registration and Login

[0908] First, a user accesses the system using a device (smartphone or PC) and enters the required information such as name, email address, and password into the new registration form. This information is sent to the server, which then creates a new user account in the database. When logging in, the user enters their email address and password from their device and sends them to the server. The server then refers to the database to perform authentication, and if authentication is successful, a session begins.

[0909] Photo upload

[0910] The user selects multiple photos taken during their trip from their device and makes an upload request. The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[0911] Photo analysis

[0912] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts metadata from the photos and analyzes information such as date, time, and location. The extracted metadata is then stored in a database by the server.

[0913] Gathering more information with conversational AI

[0914] The server uses an interactive algorithm based on the analyzed metadata to generate questions for the user. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. The user can also skip questions. The user's answers are sent from the device to the server, which stores them in a database.

[0915] Emotional regulation by emotion engine

[0916] The emotion engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, if a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion engine analyzes that information and generates emotional data. The emotional data is then sent to a server and stored in a database.

[0917] Automatic travel report generation

[0918] The server selects an appropriate video recording template based on the collected metadata, user responses, and emotion data. Based on the selected template, a detailed travel video is automatically generated. This video recording includes text, photos, anecdotes, and the user's emotions. The generated video recording is then provided to the user's device for viewing.

[0919] Information sharing and virtual travel experiences

[0920] Users can use their devices to set the generated travelogue video as public or private. The setting information is sent to the server and stored in a database. Travelogue videos set as public can be viewed by other users. The server generates a virtual travel experience based on the published travelogue and provides other users with detailed information about related places and activities.

[0921] Specific examples

[0922] For example, suppose a user uploads a photo of Tokyo Tower taken during a trip. The system analyzes the photo's metadata and automatically obtains "Tokyo Tower information." The user then inputs "I had fun at this place" into the emotion engine. Based on this information, a video travelogue is generated as a fun memory.

[0923] Example prompts for generative AI models

[0924] Upload a photo of Tokyo Tower and write a travel journal about the fun you had.

[0925] In this way, the system of the present invention allows users to easily organize travel photos, automatically create travel journals that reflect their emotions, and share them with other users.

[0926] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0927] Step 1: User registration and login

[0928] First, a user accesses the system using a terminal and enters the required information such as name, email address, and password into a new registration form. The terminal sends this information to the server, which receives the information and creates a new user account in the database. When logging in, the user enters their email address and password from the terminal and sends them to the server. The server refers to the database to perform authentication, and if authentication is successful, a session begins.

[0929] Input: User information (name, email address, password)

[0930] Output: User account creation and authentication session

[0931] Step 2: Upload a photo

[0932] The user selects multiple photos taken during their trip from their device and makes an upload request. The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[0933] Input: A photo selected by the user

[0934] Output: Photos saved in cloud storage and a record of their location

[0935] Step 3: Photo analysis

[0936] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts metadata from the photos and analyzes information such as date, time, and location. The extracted metadata is then stored in a database by the server.

[0937] Input: Photos stored in cloud storage

[0938] Output: Extracted and parsed metadata (date, time, location, etc.)

[0939] Step 4: Gathering more information with conversational AI

[0940] The server uses an interactive algorithm based on the analyzed metadata to generate questions for the user. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. The user can also skip questions. The user's answers are sent from the device to the server, which stores them in a database.

[0941] Input: Parsed metadata

[0942] Output: User details and answer questions

[0943] Step 5: Emotional regulation with the Emotion Engine

[0944] The emotion engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, if a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion engine analyzes that information and generates emotional data. The emotional data is then sent to a server and stored in a database.

[0945] Input: User's emotional expression

[0946] Output: Generated emotion data

[0947] Step 6: Automatic travel journal generation

[0948] The server selects an appropriate video recording template based on the collected metadata, user responses, and emotion data. Based on the selected template, a detailed travel video is automatically generated. This video recording includes text, photos, anecdotes, and the user's emotions. The generated video recording is then provided to the user's device.

[0949] Input: Metadata, user responses, sentiment data

[0950] Output: Automatically generated travel video

[0951] Step 7: Information sharing and virtual travel experiences

[0952] Users can use their devices to set the generated travelogue video as public or private. The setting information is sent to the server and stored in a database. Travelogue videos set as public can be viewed by other users. The server generates a virtual travel experience based on the published travelogue and provides other users with detailed information about related places and activities.

[0953] Input: Information set to public or private

[0954] Output: Published travelogues and virtual travel experiences

[0955] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0956] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0957] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0958] [Third embodiment]

[0959] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0960] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0961] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0962] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0963] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0964] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0965] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0966] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0967] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0968] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0969] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0970] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0971] This invention relates to a system that allows a user to upload multiple photos taken during a trip, analyzes the date, time, location, and detailed information from those photos, and automatically generates a travel journal. Specific embodiments for implementing this system and their program processing are described below.

[0972] User Registration and Login

[0973] A user accesses the system using a terminal and first registers, entering information such as name, email address, and password into the registration form, which the terminal then sends to the server, which receives the information and creates a new user account in the database.

[0974] When logging in, the user enters their email address and password from their device and sends them to the server. The server then refers to the database for authentication, and if successful, starts a session.

[0975] Photo upload

[0976] The user uploads multiple photos taken during their trip from their device. The device sends a request to upload the selected photos to the server, and the server stores the received photos in cloud storage. The server records the storage location of each photo in a database.

[0977] Photo analysis

[0978] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts and analyzes the photo's metadata (such as the date and time of the photo's capture, GPS information, etc.). The extracted metadata is then stored in a database by the server.

[0979] Gathering more information with conversational AI

[0980] The server uses an interactive algorithm to generate questions based on the parsed metadata. For example, questions like "Where was this photo taken?" or "What experience did you have here?" are sent to the user's device, and the user answers them. The user can also skip questions. The device sends the user's answers to the server, which stores them in a database.

[0981] Automatic travel report generation

[0982] The server selects an appropriate travelogue template based on the collected information (analysis metadata and user responses) and automatically generates a detailed travelogue, which includes text, photos, anecdotes, etc. The generated travelogue is then delivered to the user's device for viewing.

[0983] Information sharing and virtual travel experiences

[0984] Users can set their travel journals to be public or private using their devices. The setting information is sent to the server, which stores it in a database. Travel journals set to public can be viewed by other users. The server generates a virtual travel experience based on the published travel journal and provides it to other users, along with detailed information about related places and activities.

[0985] These processes allow users to easily organize their travel photos, automatically generate detailed and engaging travel journals, and share them with other users to enjoy virtual travel experiences.

[0986] The processing flow will be explained below.

[0987] Step 1:

[0988] A user accesses the system using a terminal and enters the required information, such as name, email address, and password, into the new registration form.

[0989] Step 2:

[0990] The device sends the user's input information to the server and makes a registration request.

[0991] Step 3:

[0992] The server takes the information you enter and creates a new user account in its database.

[0993] Step 4:

[0994] From next time onwards, the user will enter their email address and password into the login form on their device to request a login.

[0995] Step 5:

[0996] The terminal sends authentication information to the server, and the server performs authentication by referencing a database.

[0997] Step 6:

[0998] If the server successfully authenticates the user, it starts a session for the user and grants access rights.

[0999] Step 7:

[1000] The user uses the device to select multiple photos taken during the trip and submits an upload request.

[1001] Step 8:

[1002] The device transmits the selected photo data to the server, and the server stores the photos in cloud storage.

[1003] Step 9:

[1004] The server records the location of each photo in a database.

[1005] Step 10:

[1006] The server sequentially passes the photos stored in cloud storage to the generation algorithm.

[1007] Step 11:

[1008] The generation algorithm extracts and analyzes the photo's metadata (such as the date and time the photo was taken and GPS information).

[1009] Step 12:

[1010] The server stores the extracted metadata in a database and organizes the parsed information.

[1011] Step 13:

[1012] The server uses an interactive algorithm to generate questions for the user based on the parsed metadata.

[1013] Step 14:

[1014] The interactive algorithm generates questions and sends them to the user's device.

[1015] Step 15:

[1016] The user uses the device to answer questions posed by the conversational algorithm or to skip questions.

[1017] Step 16:

[1018] The device sends the user's answers to the server, which stores them in a database.

[1019] Step 17:

[1020] The server selects an appropriate travelogue template based on the information it collects (metadata and user responses).

[1021] Step 18:

[1022] The server fills in the information in the selected template and automatically generates a detailed travelogue.

[1023] Step 19:

[1024] The server provides the generated travelogue to the user's terminal, and the user views the travelogue.

[1025] Step 20:

[1026] Users can use their devices to set their travel journals to be public or private.

[1027] Step 21:

[1028] The terminal sends the configuration information to the server, which stores it in a database.

[1029] Step 22:

[1030] Other users can use their devices to view your published travelogue.

[1031] Step 23:

[1032] The server generates a virtual travel experience based on the published travelogue and provides it to other users.

[1033] Step 24:

[1034] The server also provides detailed information about related places and activities to support virtual travel.

[1035] These steps allow users to easily organize their travel photos, automatically generate and share detailed travel journals, and share their travel journals with other users to enjoy virtual travel experiences.

[1036] Example 1

[1037] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1038] Efficiently organizing the many images taken during a trip and generating a detailed travelogue based on them is a time-consuming and labor-intensive process. There is also a need for a method to share travelogues with other users and provide them with virtual travel experiences. However, existing systems lack the functionality to allow users to easily analyze image metadata and automatically generate detailed travelogues. Furthermore, they lack an interactive algorithm that generates questions based on user answers, and an effective means to provide virtual travel experiences.

[1039] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1040] In this invention, the server includes a means for users to upload multiple images via their terminals, a means for the server to store the images in cloud storage, a means for a generation algorithm to extract and analyze metadata from the images, a means for an interactive algorithm to ask the user detailed information and store the answers, a means for automatically generating a detailed travelogue based on the generation algorithm and the user's answers, and a means for enabling the automatically generated travelogue to be shared among users. This allows users to easily upload images taken during their trip and automatically generate and share a detailed travelogue based on the images. A virtual travel experience is also provided, allowing other users to share their travelogues and enjoy virtual experiences.

[1041] "User" means the person who uses the system to upload images or input information.

[1042] A "terminal" is an electronic device such as a computer or smartphone used by a user.

[1043] "Images" are photo files taken by users during their travels and uploaded to the system.

[1044] The "server" refers to a central management system that stores images and analyzes metadata.

[1045] "Cloud storage" is an external service for storing and managing data via the Internet.

[1046] "Generation algorithm" refers to software that extracts and analyzes image metadata.

[1047] "Metadata" is additional information including the date and time an image was taken and GPS information.

[1048] An "interactive algorithm" is software that generates questions to gather more information from users and stores their answers.

[1049] A "travel journal" is a written record generated based on images and information provided by the user.

[1050] "Virtual Travel Experience" is a feature that allows other users to virtually experience travel based on automatically generated travelogues.

[1051] This invention relates to a system that allows a user to upload multiple images taken during a trip, analyzes the date, time, location, and detailed information from those images, and automatically generates a travelogue. Specific embodiments for implementing this system are described below.

[1052] User Registration and Login

[1053] A user accesses the system using a terminal and first registers. The user enters their name, email address, and password into a registration form, and the terminal sends this information to the server. The server receives the entered information and creates a new user account in the database. When the user logs in, they enter their email address and password and send them to the server. The server refers to the database to authenticate them, and if successful, starts a session.

[1054] Specific behavior:

[1055] The user fills in the required information in the registration form and clicks the "Register" button.

[1056] The device sends the input to the server.

[1057] The server creates a new user account in the database.

[1058] Photo upload

[1059] The user uploads multiple photos taken during their trip from their device. The device sends a request to upload the selected photos to the server, and the server stores the received photos in cloud storage such as Google Cloud Storage. The server records the storage location of each photo in a database.

[1060] Specific behavior:

[1061] The user selects multiple photos from a file folder and clicks the "Upload" button.

[1062] The terminal transmits the selected photo file to the server.

[1063] The server stores the photo in Google Cloud Storage and records the URL in a database.

[1064] Photo analysis

[1065] The server sequentially retrieves photos stored in cloud storage and extracts the photo metadata (date and time of shooting, GPS information, etc.) using image analysis libraries such as OpenCV and TensorFlow. The extracted metadata is then stored in a database by the server.

[1066] Specific behavior:

[1067] The server retrieves the photos from the cloud storage.

[1068] The server uses an image analysis library to extract the metadata.

[1069] The extracted metadata is stored in a database.

[1070] Gathering more information with conversational AI

[1071] The server uses an interactive algorithm to generate questions based on the parsed metadata, sends the generated questions to the user's device, and the user can answer or skip the questions. The device then sends the user's answers to the server, which stores them in a database.

[1072] Specific behavior:

[1073] The server generates questions based on the metadata.

[1074] Example questions: "Where was this photo taken?"

[1075] The device displays a question to the user.

[1076] The user enters the answer and the device sends the answer to the server.

[1077] Automatic travel report generation

[1078] The server selects an appropriate travelogue template based on the collected information (analysis metadata and user responses), and automatically generates a detailed travelogue using a generative AI model such as GPT-3. The generated travelogue is then sent to the user's device for viewing.

[1079] Example prompt sentence:

[1080] "These are travelogues to Tokyo. Please generate a detailed travelogue based on the information below."

[1081] Specific behavior:

[1082] The server sends a prompt to the generative AI model.

[1083] The generated travelogue is sent to the user's device.

[1084] Information sharing and virtual travel experiences

[1085] Users set their travel journals to be public or private and send the setting information to the server. Travel journals that are public can be viewed by other users. The server generates a virtual travel experience based on the published travel journal and provides it to other users.

[1086] Specific behavior:

[1087] Users can set their travel journals to be public or private.

[1088] The server acquires publicly set travelogues and generates a virtual travel experience.

[1089] By implementing this system, users can easily organize their travel photos, automatically generate detailed and attractive travel journals, and share them with other users to enjoy virtual travel experiences.

[1090] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1091] Step 1: User Registration

[1092] The user enters their name, email address, and password in plain text.

[1093] The device sends this input information to the server in JSON format.

[1094] The server stores the received information in its database as a new user account.

[1095] Enter your name, email address, and password.

[1096] Data processing: Convert input information into JSON format.

[1097] Output: A new user account is created in the database.

[1098] Specific behavior:

[1099] The user enters "Yamada Taro", "taro@example.com", and a password in the registration form and clicks the "Register" button.

[1100] The device sends this information to the server in JSON format.

[1101] The server receives the information and creates a new user account in the database.

[1102] Step 2: Log in

[1103] The user enters their email address and password in plain text.

[1104] The device sends this input information to the server in JSON format.

[1105] The server refers to the database and performs authentication. If authentication is successful, a session is started.

[1106] Enter your email address and password.

[1107] Data calculation: Matching input information with existing information in a database.

[1108] Output: A session ID is generated.

[1109] Specific behavior:

[1110] The user enters "taro@example.com" and their password in the login form and clicks the "Login" button.

[1111] The device sends this information to the server in JSON format.

[1112] The server queries the database, and since authentication is successful, generates a session ID.

[1113] Step 3: Upload your photos

[1114] The user selects multiple photos taken during their trip from their device. The selection is made in file format.

[1115] The device sends the selected photo file to the server via an HTTP POST request.

[1116] The server stores the received photos in Google Cloud Storage and records their URLs in the database.

[1117] Input: Travel photo files.

[1118] Data processing: Convert the photo file into HTTP POST request format.

[1119] Output: The photo is saved to Google Cloud Storage and its URL is recorded in the database.

[1120] Specific behavior:

[1121] The user selects "IMG_001.jpg" to "IMG_010.jpg" from the file folder and clicks the "Upload" button.

[1122] The device sends the selected photo file to the server via an HTTP POST request.

[1123] The server stores the received photos in Google Cloud Storage and records their URLs in the database.

[1124] Step 4: Photo analysis

[1125] The server retrieves photos sequentially from Google Cloud Storage.

[1126] The server uses image analysis libraries such as OpenCV and TensorFlow to extract photo metadata (such as the date and time the photo was taken, GPS information, etc.).

[1127] The server stores the extracted metadata in a database.

[1128] Input: Google Cloud Storage photo URL.

[1129] Data Computation: Extraction of metadata using image analysis libraries.

[1130] Output: The extracted metadata is stored in a database.

[1131] Specific behavior:

[1132] The server retrieves "IMG_001.jpg" from Google Cloud Storage.

[1133] The server analyzes the EXIF ​​information of the photo using OpenCV and extracts the date and time of the photo and GPS information.

[1134] The extracted metadata is stored in a database.

[1135] Step 5: Gathering more information with conversational AI

[1136] The server generates questions based on the parsed metadata, using a generative AI model (e.g., GPT-3).

[1137] The device displays the generated question to the user, who can then enter an answer or skip.

[1138] The device sends the user's answers to the server, which stores them in a database.

[1139] Input: Metadata.

[1140] Data Computing: Generate questions and store answers using generative AI models.

[1141] Output: The generated questions and the user's answers are stored in a database.

[1142] Specific behavior:

[1143] Based on the metadata, the server sends the question "Where was this photo taken?" as a prompt to the generative AI model and receives the question as an answer.

[1144] The device displays a question to the user.

[1145] The user answers "Tokyo Tower," and the device sends the answer to the server.

[1146] The server stores the answer in a database.

[1147] Step 6: Automatic travel journal generation

[1148] The server selects an appropriate travelogue template based on the collected metadata and the user's answers.

[1149] The server automatically generates a detailed travelogue using a generative AI model (e.g., GPT-3).

[1150] The server sends the generated travelogue to the user's terminal.

[1151] Input: Metadata, user answers.

[1152] Data computation: Generating travel journals using generative AI models.

[1153] Output: The automatically generated travelogue is sent to the user's device.

[1154] Specific behavior:

[1155] The server converts the metadata and answers into prompts and sends them to the generative AI model.

[1156] The generated travelogue is sent to the user's device and can be viewed by the user.

[1157] Step 7: Information sharing and virtual travel experiences

[1158] The user sets the travel journal to be public or private and sends it from the device to the server.

[1159] The server stores the configuration information in a database.

[1160] The server generates a virtual travel experience based on the published travelogue and provides it to other users.

[1161] Input: Public / Private settings.

[1162] Data calculation: Converting published travel journal data into virtual experience content.

[1163] Output: The publicly set travelogue is shared with other users, providing them with a virtual travel experience.

[1164] Specific behavior:

[1165] The user sets whether their travel journal is public or private, and sends the setting information from their device to the server.

[1166] The server stores the configuration information in a database.

[1167] The server acquires the published travelogue data, generates a virtual travel experience, and provides it to other users.

[1168] The above is the specific processing flow of this system and the operation at each step.

[1169] (Application example 1)

[1170] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1171] In today's world, many users find it time-consuming to simply organize photos taken during a trip and record their experiences in a detailed travel journal. It's also difficult to share those travel journals with others or provide virtual travel experiences. This often results in valuable travel experiences remaining buried. The present invention aims to solve these problems by providing a system that automatically and efficiently generates travel journals based on photos taken by users and allows them to publish and share them in a virtual store.

[1172] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1173] In this invention, the server includes: a means for a user to upload multiple images via a terminal; a means for the server to store the images in cloud storage; a means for a generation algorithm to extract and analyze metadata from the images; a means for an interactive algorithm to ask the user detailed information and store the answers; a means for automatically generating detailed records based on the generation algorithm and the user's answers; a means for enabling the automatically generated records to be shared among users; a means for providing a virtual experience based on the collected information and the generated records; and a means for publishing the generated records in a virtual store. This allows users to efficiently organize their travel photos, automatically generate and publish attractive travel journals, and share and enjoy them with other users.

[1174] "User" means any person or entity that accesses the system, uploads multiple images, and inputs information to generate detailed records from those images.

[1175] "Terminal" means the hardware device used by a User to access the System and upload images or enter information.

[1176] "Server" refers to the back-end computing resources used to store images received from users and run generative and interactive algorithms.

[1177] "Cloud storage" refers to an external storage system that allows a server to safely store image data received via the Internet.

[1178] "Generation Algorithm" refers to the software process used to extract and analyze metadata from uploaded images and automatically generate detailed records.

[1179] "Metadata" refers to data that contains additional information specific to an image, such as the date and time the image was taken or GPS information.

[1180] "Interactive algorithm" refers to the software process by which a system generates questions to gather more information from users and stores their answers.

[1181] "Detailed Records" refers to travel journals and other documents that are automatically generated based on images and information provided by users.

[1182] "Virtual Store" refers to an online platform for publishing and sharing detailed user-generated records with other users.

[1183] A "virtual experience" is an experience that allows other users to take a virtual journey based on a publicly detailed record.

[1184] This invention relates to a system that allows users to upload multiple images taken during their trip, analyzes the date, time, location, and detailed information from those images, and automatically generates a detailed record. Specific embodiments for implementing this system and their program processing are described below.

[1185] User Registration and Login

[1186] A user accesses the system using a terminal and first registers. They enter information such as their name, email address, and password into the registration form, and the terminal sends this information to the server. The server receives the entered information and creates a new user account in its database. When logging in, the user enters their email address and password from the terminal and sends them to the server. The server refers to the database to perform authentication, and if successful, starts a session.

[1187] Photo upload

[1188] The user uploads multiple photos taken during their trip from their device. The device sends a request to upload the selected photos to the server, which then stores the photos in cloud storage. The server records the location of each photo in a database.

[1189] Photo analysis

[1190] The server sequentially passes photos stored in the cloud storage to a generation algorithm, which extracts and analyzes the photo's metadata (such as the date and time of the photo's capture, GPS information, etc.) The extracted metadata is then stored in a database by the server.

[1191] Gathering more information with conversational AI

[1192] The server uses an interactive algorithm to generate questions based on the parsed metadata. For example, questions like "Where was this photo taken?" or "What experience did you have here?" are sent to the user's device, and the user answers them. The user can also skip questions. The device sends the user's answers to the server, which stores them in a database.

[1193] Examples of specific prompts include:

[1194] 1. Question: "Where was this photo taken?"

[1195] User Answer: "This photo was taken at Kiyomizu-dera Temple in Kyoto."

[1196] 2. Question: "What was your experience here?"

[1197] User Answer: "I enjoyed visiting tourist spots and eating my way around."

[1198] Detailed automatic recording

[1199] The server selects an appropriate template based on the collected information (analysis metadata and user responses) and automatically generates a detailed record, which includes text, photos, anecdotes, etc. The generated record is then delivered to the user's device for viewing.

[1200] Virtual Experience and Information Sharing

[1201] Users can use their devices to set their detailed records to public or private. The setting information is sent to the server, which stores it in a database. Publicly set detailed records can be viewed by other users. The server generates a virtual experience based on the publicly set detailed records and provides it to other users, along with detailed information about related locations and activities.

[1202] The system is implemented using the following hardware and software:

[1203] Server: Cloud server, database (e.g., Amazon RDS)

[1204] Image storage: Cloud storage (e.g. AWS S3)

[1205] Generative AI models: Libraries for running generative algorithms, such as TensorFlow and PyTorch

[1206] This allows users to efficiently organize their travel photos, automatically generate and publish compelling travel journals, and share and enjoy them with other users.

[1207] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1208] Step 1: User registration and login

[1209] A user accesses the system using a terminal, enters their name, email address, and password, and submits it. The server receives this information and creates a new user account in the database. When logging in, the user enters their email address and password. The server uses this information to look up their database and authenticates them, and if successful, starts a session.

[1210] Input: User's name, email address, password

[1211] Output: New user account created and authenticated.

[1212] Step 2: Upload a photo

[1213] Users select multiple travel photos from their devices and send an upload request to the server, which then stores the photos in cloud storage and records their location in a database.

[1214] Input: Multiple photos taken by the user

[1215] Output: Save photos to cloud storage and record their location in a database

[1216] Step 3: Photo analysis

[1217] The server passes the photos stored in the cloud storage to a generation algorithm, which extracts and analyzes the photo's metadata (date and time of the photo, GPS information), which is then stored in a database by the server.

[1218] Input: Photos stored in cloud storage

[1219] Output: Extracted metadata stored in a database

[1220] Step 4: Gathering more information with conversational AI

[1221] The server uses an interactive algorithm to generate questions based on the parsed metadata. The generated questions are sent to the user's device, and the user answers them. The device then sends the user's answers to the server, which stores them in a database.

[1222] Input: extracted metadata, user answers

[1223] Output: Generated questions, saving user answers to database

[1224] Examples of specific prompts include:

[1225] 1. Question: "Where was this photo taken?"

[1226] User Answer: "This photo was taken at Kiyomizu-dera Temple in Kyoto."

[1227] 2. Question: "What was your experience here?"

[1228] User Answer: "I enjoyed visiting tourist spots and eating my way around."

[1229] Step 5: Automatically generate detailed records

[1230] The server automatically generates a detailed record based on the generation algorithm and the user's answers. The generated record includes text, photos, anecdotes, etc. The server then provides the generated record to the user's device.

[1231] Input: Analysis metadata, user answers

[1232] Output: Auto-generated detailed transcript

[1233] Step 6: Virtual Experience and Information Sharing

[1234] Users can use their devices to set their detailed records to public or private. The server stores this setting information in a database. Publicly set detailed records can be viewed by other users. The server generates a virtual experience based on the publicly set detailed records and provides it to other users.

[1235] Input: User's public / private settings

[1236] Output: Publishing detailed records and generating virtual experiences

[1237] These processing flows allow users to efficiently organize photos taken during their trip, automatically generate and publish detailed and engaging travel journals, and share their virtual experiences with other users.

[1238] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1239] The present invention relates to a system that allows users to upload multiple photos taken during their trip, analyzes the date, time, location, and detailed information from those photos, and automatically generates a travel diary. Furthermore, by combining this system with an emotion engine, the system can recognize the user's emotions and reflect them in the travel diary. Specific embodiments of the present invention and the processing of the program are described below.

[1240] User Registration and Login

[1241] A user accesses the system using a terminal and enters the required information such as name, email address, password, etc. into a new registration form. The terminal sends this information to the server, which receives it and creates a new user account in the database.

[1242] When logging in, the user enters their email address and password from their device and sends them to the server. The server then refers to the database for authentication, and if authentication is successful, a session begins.

[1243] Photo upload

[1244] The user selects multiple photos taken during their trip using their device and makes an upload request. The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[1245] Photo analysis

[1246] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts metadata from the photos and analyzes information such as date, time, and location. The extracted metadata is then stored in a database by the server.

[1247] Gathering more information with conversational AI

[1248] The server uses an interactive algorithm to generate questions for the user based on the parsed metadata. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. The user can also skip questions. The device sends the answers to the server, which stores them in a database.

[1249] Emotional regulation by emotion engine

[1250] The emotion engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, when a user inputs an emotion expression such as "I had fun" or "I'm a little tired," the emotion engine analyzes the information and generates emotion data. The emotion data is then sent to the server and stored in a database.

[1251] Automatic travel report generation

[1252] The server selects an appropriate travelogue template based on the collected metadata, user responses, and emotion data. A detailed travelogue is automatically generated based on the selected template. This travelogue includes text, photos, anecdotes, and the user's emotions. The generated travelogue is then provided to the user's device for viewing.

[1253] Information sharing and virtual travel experiences

[1254] Users can set their travel journals to be public or private using their devices. The setting information is sent to the server and stored in a database. Travel journals set to public can be viewed by other users. The server generates a virtual travel experience based on the published travel journal and provides other users with detailed information on related places and activities.

[1255] For example, if a user uploads photos taken during a trip and inputs the emotion "It was fun" through the emotion engine, the system will analyze this information and reflect it in the travelogue as a "fun memory." In addition, when other users view this travelogue, content will be provided that allows them to feel a "fun atmosphere" as if they are traveling virtually.

[1256] This allows the system of the present invention to not only allow users to easily organize their travel photos and automatically generate detailed travel journals, but also to create records that reflect the user's emotions and share them with other users. Furthermore, other users can gain information and experiences about new travel destinations through virtual travel experiences.

[1257] The processing flow will be explained below.

[1258] Step 1:

[1259] A user accesses the system using a terminal and enters the required information, such as name, email address, and password, into the new registration form.

[1260] Step 2:

[1261] The terminal sends the entered user information to the server, which then creates a new user account in the database.

[1262] Step 3:

[1263] The user uses a device to enter their email address and password into the login form to request a login.

[1264] Step 4:

[1265] The terminal sends authentication information to the server, which then refers to the database to authenticate the user. If authentication is successful, the server starts a session for the user.

[1266] Step 5:

[1267] A user uses a device to select multiple travel photos and make an upload request.

[1268] Step 6:

[1269] The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[1270] Step 7:

[1271] The server sequentially passes the photos stored in cloud storage to the generation algorithm.

[1272] Step 8:

[1273] The generation algorithm extracts and analyzes the metadata of the photo (such as the date and time of the photo being taken, GPS information, etc.), and the extracted metadata is stored in a database by the server.

[1274] Step 9:

[1275] The server uses an interactive algorithm to generate questions for the user based on the parsed metadata.

[1276] Step 10:

[1277] The conversational algorithm generates questions and sends them to the user's device, such as "Where was this photo taken?" or "What kind of experience did you have here?"

[1278] Step 11:

[1279] The user uses the device to answer questions posed by the conversational algorithm or to skip questions.

[1280] Step 12:

[1281] The device sends the user's answers to the server, which stores them in a database.

[1282] Step 13:

[1283] The conversational algorithm uses an emotion engine to recognize the user's emotions. For example, if the user answers "I had fun" or "I'm a little tired," the emotion will be recognized.

[1284] Step 14:

[1285] The emotion engine generates the recognized emotion data and sends it to the server, which stores the emotion data in a database.

[1286] Step 15:

[1287] The server selects an appropriate travel diary template based on the collected metadata, user responses, and sentiment data.

[1288] Step 16:

[1289] The server automatically generates a detailed travelogue by filling in the selected template with information, including text, photos, anecdotes, and the user's emotions.

[1290] Step 17:

[1291] The server provides the generated travelogue to the user's terminal, and the user views the travelogue.

[1292] Step 18:

[1293] Users can use their devices to set their travel journals to be public or private.

[1294] Step 19:

[1295] The terminal sends the configuration information to the server, which stores the configuration information in a database.

[1296] Step 20:

[1297] Other users can use their devices to view your published travelogue.

[1298] Step 21:

[1299] The server generates a virtual travel experience based on the published travelogue. Emotional data is also taken into consideration, and content adapted to the user's emotions is provided in the virtual travel experience.

[1300] Step 22:

[1301] The server provides detailed information about places and activities related to the virtual travel experience, and helps other users gain information and experiences about new travel destinations through the travelogue.

[1302] These steps allow users to easily organize their travel photos, automatically generate and share detailed and emotional travel journals, and allow other users to experience virtual travel through the journals.

[1303] Example 2

[1304] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1305] Conventional methods require users to organize multiple photos taken during a trip and automatically generate a travelogue by analyzing the date, time, location, and detailed information from those photos. This requires a lot of effort and effort. It is also difficult to create a travelogue that reflects emotions, and it is not easy to share or provide a virtual travel experience. There is a need for a system that solves these problems and enables efficient automatic generation and sharing of travelogues that reflect emotions.

[1306] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1307] In this invention, the server includes means for a user to upload multiple images via a device, means for the server to store the images in a cloud storage device, means for a generation algorithm to extract and analyze metadata for the images, means for an interactive algorithm to ask the user detailed information and store the responses, means for an emotion recognition engine to recognize the user's emotions and analyze and store the emotion data, means for automatically generating detailed travelogues based on the generation algorithm, the user's responses, and the emotion data, and means for enabling the automatically generated travelogues to be shared among users, thereby enabling users to efficiently organize photos taken during their trips, automatically generate detailed travelogues, and create travelogues that reflect their emotions and share them with other users.

[1308] "User" refers to any individual or entity that uses the System to generate travel journals and upload photos.

[1309] "Device" refers to the hardware used by a user, such as a computer, smartphone, or tablet.

[1310] "Images" refers to photographic data taken by users during their travels and uploaded to the system.

[1311] "Server" refers to a computer system that controls the entire system and exchanges data with users.

[1312] "Cloud storage" refers to a storage service for storing data via the Internet.

[1313] "Generation algorithm" refers to a program for extracting and analyzing image metadata.

[1314] "Metadata" refers to additional information contained in an image, such as the date, time, location, and shooting conditions.

[1315] An "interactive algorithm" is a program designed to ask users detailed questions and collect their answers.

[1316] "Emotion recognition engine" refers to a system for analyzing and storing user emotional data.

[1317] "Emotional data" refers to information analyzed by an emotion recognition engine based on emotional expressions entered by the user.

[1318] "Records" refers to detailed travel journals that are automatically generated based on generative algorithms, user responses, and sentiment data.

[1319] "Virtual travel experience" refers to content that allows other users to experience the atmosphere and information of a trip based on publicly available records.

[1320] "Publicity Settings" refers to the settings options that allow a User to make a Recording that the User generates available to other Users.

[1321] The present invention relates to a system that allows users to upload multiple photos taken during their trip, analyzes the date, time, location, and detailed information from those photos, and automatically generates a travel diary. Furthermore, by combining this system with an emotion recognition engine, the system can recognize the user's emotions and reflect them in the travel diary. Specific embodiments of the present invention and the processing of the program will be described below.

[1322] User Registration and Login

[1323] A user accesses the system using a terminal and enters the required information (such as name, email address, and password) in a new registration form. The terminal sends this information to the server, which then stores it in a database and creates a new user account. When logging in, the user enters their email address and password from the terminal and sends them to the server, which then references the database to perform authentication and starts a session if authentication is successful.

[1324] Photo upload

[1325] A user uses a device to select multiple photos taken during a trip and request uploading. The device sends the selected photo data to a server, which then stores the received photos in a cloud storage device and records the storage location of each photo in a database.

[1326] Photo analysis

[1327] The server processes the photos stored in the cloud storage device sequentially. Specifically, it uses a generative algorithm (e.g., Google Cloud Vision API) to extract and analyze the photo metadata (date, time, location, etc.). The server stores the analyzed metadata in a database.

[1328] Gathering more information with conversational AI

[1329] Based on the analyzed metadata, the server uses a conversational algorithm (e.g., GPT-3) to generate questions for the user. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. Prompt sentences such as "Where was this photo taken?" or "How did you feel when you took the photo?" may also be used. The device sends the answers to the server, which stores them in a database. The user can also skip questions.

[1330] Emotional regulation by emotion engine

[1331] The emotion recognition engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, when a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion recognition engine analyzes it and generates emotional data. The emotional data is then sent to the server and stored in a database.

[1332] Automatic travel report generation

[1333] The server selects an appropriate travelogue template based on the collected metadata, the user's responses, and sentiment data. For example, it generates a detailed travelogue based on information such as "I took the photo at the top of the Eiffel Tower. The view was amazing." The generated travelogue is then sent to the user's device for viewing.

[1334] Information sharing and virtual travel experiences

[1335] The user sets the travel journal to be public or private using the device. This setting information is sent to the server and stored in a database. The server makes publicly set travel journals available to other users. The server also generates a virtual travel experience based on the published travel journal and provides other users with detailed information on related places and activities.

[1336] In this way, the system of the present invention allows users to efficiently organize their travel photos, automatically generate detailed travel journals, create emotional records, and share them with other users, allowing them to gain information and experiences about new travel destinations.

[1337] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1338] Step 1: User registration and login

[1339] A user accesses the system using a terminal and enters the required information (name, email address, password, etc.) into a new registration form. This becomes input data. The terminal sends this information to the server. The server stores the received information in a database and creates a new user account. This becomes output data. When logging in, the email address and password entered by the user are sent from the terminal to the server. The server refers to the database to perform authentication, and if authentication is successful, a session is started. This allows the user to access the system.

[1340] Step 2: Upload a photo

[1341] A user uses a device to select multiple photos taken during a trip and make an upload request. This is the input data. The device sends the selected photo data to the server. The server stores the received photos in cloud storage and records the storage location of each photo in a database. This is the output data. The photos are then stored in the system.

[1342] Step 3: Photo analysis

[1343] The server sequentially passes the photos stored in the cloud storage device to a generation algorithm (e.g., Google Cloud Vision API). This becomes the input data. The generation algorithm extracts and analyzes the photo's metadata (date, time, location, etc.). The server stores the analyzed metadata in a database. This becomes the output data. As a result, detailed information about the photo is extracted and stored in the database.

[1344] Step 4: Gathering more information with conversational AI

[1345] Based on the analyzed metadata, the server uses an interactive algorithm (e.g., GPT-3) to generate questions for the user. This becomes the input data. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. This becomes the output data. The server sends the generated questions to the user's device, and the user answers the questions. The device sends the answers to the server, and the server stores them in a database. The user can also skip questions, allowing more information to be collected.

[1346] Step 5: Emotional regulation with the Emotion Engine

[1347] The emotion recognition engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. This becomes the input data. For example, when a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion recognition engine analyzes it and generates emotional data. The emotional data is sent to the server and stored in a database. This becomes the output data. This records the user's emotions.

[1348] Step 6: Automatic travel journal generation

[1349] The server selects an appropriate travelogue template based on the collected metadata, user responses, and emotional data. This becomes the input data. A detailed travelogue is generated by incorporating the metadata, episodes, and emotional information into the template. The generated travelogue is then provided to the user's device. This becomes the output data. The user can then view the detailed travelogue.

[1350] Step 7: Information sharing and virtual travel experiences

[1351] The user uses their device to set their travel journal to be public or private. This becomes the input data. The setting information is sent to the server and stored in a database. The server then sets the public travel journal so that other users can view it. This becomes the output data. The server also generates virtual travel experience content based on the published travel journal and provides other users with detailed information about related places and activities. This allows other users to gain information and experiences about new travel destinations.

[1352] (Application example 2)

[1353] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1354] Conventional travel journal creation systems require users to manually organize photos and detailed information to create a travel journal, a time-consuming and labor-intensive process. It is also difficult to reflect the user's emotions, making it difficult to gain emotional empathy when sharing with other users. Furthermore, automatic travel journal generation in video format is limited, making it difficult to provide an effective visual experience.

[1355] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1356] In this invention, the server includes a means for a user to upload multiple images via a device, a means for the server to store the images in cloud storage, a means for a generation algorithm to extract and analyze metadata of the images, a means for an interactive algorithm to ask the user detailed information and store the answers, a means for analyzing the user's emotions in real time using an emotion engine to generate emotion data, a means for automatically generating detailed video recordings based on the generation algorithm, the user's answers, and the emotion data, and a means for enabling the automatically generated recordings to be shared among users. This allows users to easily upload photos, automatically generate emotionally-reflecting travel journals in video format, and share them with other users.

[1357] "User" means a person who uses a digital device to access the system and create and share travel journals.

[1358] "Device" refers to electronic devices with communication capabilities, such as computers, smartphones, and tablets.

[1359] "Images" refers to visual data such as photographs and illustrations taken by users during their travels.

[1360] A "server" is a computer system within a network that receives, processes, and stores data submitted by users.

[1361] "Cloud storage" refers to a service that stores data on remote servers accessible via the Internet.

[1362] A "generative algorithm" is a computational method for automatically generating new content based on input data.

[1363] "Metadata" refers to additional information that accompanies an image, such as the date, time, location, and camera settings.

[1364] An "interactive algorithm" is a program that collects information through dialogue with the user and generates appropriate questions.

[1365] An "emotion engine" is a program that analyzes emotions from user input and actions and generates them as data.

[1366] "Emotion data" is data that indicates the user's emotional state, obtained as a result of analysis by the emotion engine.

[1367] "Video recording" refers to content that integrates images, audio, and text for dynamic playback.

[1368] "Automatically Generated Records" refers to documents or media content that are automatically generated based on data collected by the system.

[1369] "Shareable" means that the records created are viewable by other users or on public platforms, allowing for the widespread dissemination of information.

[1370] A "virtual travel experience" refers to a digital simulation that allows users to experience a place through sight, sound, and other senses without actually visiting the place.

[1371] The present invention provides a system for automatically generating travel video that reflects emotions based on travel photos taken by a user, and sharing the video with other users. Specific embodiments are described below.

[1372] User Registration and Login

[1373] First, a user accesses the system using a device (smartphone or PC) and enters the required information such as name, email address, and password into the new registration form. This information is sent to the server, which then creates a new user account in the database. When logging in, the user enters their email address and password from their device and sends them to the server. The server then refers to the database to perform authentication, and if authentication is successful, a session begins.

[1374] Photo upload

[1375] The user selects multiple photos taken during their trip from their device and makes an upload request. The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[1376] Photo analysis

[1377] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts metadata from the photos and analyzes information such as date, time, and location. The extracted metadata is then stored in a database by the server.

[1378] Gathering more information with conversational AI

[1379] The server uses an interactive algorithm based on the analyzed metadata to generate questions for the user. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. The user can also skip questions. The user's answers are sent from the device to the server, which stores them in a database.

[1380] Emotional regulation by emotion engine

[1381] The emotion engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, if a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion engine analyzes that information and generates emotional data. The emotional data is then sent to a server and stored in a database.

[1382] Automatic travel report generation

[1383] The server selects an appropriate video recording template based on the collected metadata, user responses, and emotion data. Based on the selected template, a detailed travel video is automatically generated. This video recording includes text, photos, anecdotes, and the user's emotions. The generated video recording is then provided to the user's device for viewing.

[1384] Information sharing and virtual travel experiences

[1385] Users can use their devices to set the generated travelogue video as public or private. The setting information is sent to the server and stored in a database. Travelogue videos set as public can be viewed by other users. The server generates a virtual travel experience based on the published travelogue and provides other users with detailed information about related places and activities.

[1386] Specific examples

[1387] For example, suppose a user uploads a photo of Tokyo Tower taken during a trip. The system analyzes the photo's metadata and automatically obtains "Tokyo Tower information." The user then inputs "I had fun at this place" into the emotion engine. Based on this information, a video travelogue is generated as a fun memory.

[1388] Example prompts for generative AI models

[1389] Upload a photo of Tokyo Tower and write a travel journal about the fun you had.

[1390] In this way, the system of the present invention allows users to easily organize travel photos, automatically create travel journals that reflect their emotions, and share them with other users.

[1391] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1392] Step 1: User registration and login

[1393] First, a user accesses the system using a terminal and enters the required information such as name, email address, and password into a new registration form. The terminal sends this information to the server, which receives the information and creates a new user account in the database. When logging in, the user enters their email address and password from the terminal and sends them to the server. The server refers to the database to perform authentication, and if authentication is successful, a session begins.

[1394] Input: User information (name, email address, password)

[1395] Output: User account creation and authentication session

[1396] Step 2: Upload a photo

[1397] The user selects multiple photos taken during their trip from their device and makes an upload request. The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[1398] Input: A photo selected by the user

[1399] Output: Photos saved in cloud storage and a record of their location

[1400] Step 3: Photo analysis

[1401] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts metadata from the photos and analyzes information such as date, time, and location. The extracted metadata is then stored in a database by the server.

[1402] Input: Photos stored in cloud storage

[1403] Output: Extracted and parsed metadata (date, time, location, etc.)

[1404] Step 4: Gathering more information with conversational AI

[1405] The server uses an interactive algorithm based on the analyzed metadata to generate questions for the user. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. The user can also skip questions. The user's answers are sent from the device to the server, which stores them in a database.

[1406] Input: Parsed metadata

[1407] Output: User details and answer questions

[1408] Step 5: Emotional regulation with the Emotion Engine

[1409] The emotion engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, if a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion engine analyzes that information and generates emotional data. The emotional data is then sent to a server and stored in a database.

[1410] Input: User's emotional expression

[1411] Output: Generated emotion data

[1412] Step 6: Automatic travel journal generation

[1413] The server selects an appropriate video recording template based on the collected metadata, user responses, and emotion data. Based on the selected template, a detailed travel video is automatically generated. This video recording includes text, photos, anecdotes, and the user's emotions. The generated video recording is then provided to the user's device.

[1414] Input: Metadata, user responses, sentiment data

[1415] Output: Automatically generated travel video

[1416] Step 7: Information sharing and virtual travel experiences

[1417] Users can use their devices to set the generated travelogue video as public or private. The setting information is sent to the server and stored in a database. Travelogue videos set as public can be viewed by other users. The server generates a virtual travel experience based on the published travelogue and provides other users with detailed information about related places and activities.

[1418] Input: Information set to public or private

[1419] Output: Published travelogues and virtual travel experiences

[1420] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1421] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1422] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1423] [Fourth embodiment]

[1424] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1425] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1426] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1427] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1428] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1429] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1430] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1431] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1432] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1433] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1434] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1435] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1436] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1437] This invention relates to a system that allows a user to upload multiple photos taken during a trip, analyzes the date, time, location, and detailed information from those photos, and automatically generates a travel journal. Specific embodiments for implementing this system and their program processing are described below.

[1438] User Registration and Login

[1439] A user accesses the system using a terminal and first registers, entering information such as name, email address, and password into the registration form, which the terminal then sends to the server, which receives the information and creates a new user account in the database.

[1440] When logging in, the user enters their email address and password from their device and sends them to the server. The server then refers to the database for authentication, and if successful, starts a session.

[1441] Photo upload

[1442] The user uploads multiple photos taken during their trip from their device. The device sends a request to upload the selected photos to the server, and the server stores the received photos in cloud storage. The server records the storage location of each photo in a database.

[1443] Photo analysis

[1444] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts and analyzes the photo's metadata (such as the date and time of the photo's capture, GPS information, etc.). The extracted metadata is then stored in a database by the server.

[1445] Gathering more information with conversational AI

[1446] The server uses an interactive algorithm to generate questions based on the parsed metadata. For example, questions like "Where was this photo taken?" or "What experience did you have here?" are sent to the user's device, and the user answers them. The user can also skip questions. The device sends the user's answers to the server, which stores them in a database.

[1447] Automatic travel report generation

[1448] The server selects an appropriate travelogue template based on the collected information (analysis metadata and user responses) and automatically generates a detailed travelogue, which includes text, photos, anecdotes, etc. The generated travelogue is then delivered to the user's device for viewing.

[1449] Information sharing and virtual travel experiences

[1450] Users can set their travel journals to be public or private using their devices. The setting information is sent to the server, which stores it in a database. Travel journals set to public can be viewed by other users. The server generates a virtual travel experience based on the published travel journal and provides it to other users, along with detailed information about related places and activities.

[1451] These processes allow users to easily organize their travel photos, automatically generate detailed and engaging travel journals, and share them with other users to enjoy virtual travel experiences.

[1452] The processing flow will be explained below.

[1453] Step 1:

[1454] A user accesses the system using a terminal and enters the required information, such as name, email address, and password, into the new registration form.

[1455] Step 2:

[1456] The device sends the user's input information to the server and makes a registration request.

[1457] Step 3:

[1458] The server takes the information you enter and creates a new user account in its database.

[1459] Step 4:

[1460] From next time onwards, the user will enter their email address and password into the login form on their device to request a login.

[1461] Step 5:

[1462] The terminal sends authentication information to the server, and the server performs authentication by referencing a database.

[1463] Step 6:

[1464] If the server successfully authenticates the user, it starts a session for the user and grants access rights.

[1465] Step 7:

[1466] The user uses the device to select multiple photos taken during the trip and submits an upload request.

[1467] Step 8:

[1468] The device transmits the selected photo data to the server, and the server stores the photos in cloud storage.

[1469] Step 9:

[1470] The server records the location of each photo in a database.

[1471] Step 10:

[1472] The server sequentially passes the photos stored in cloud storage to the generation algorithm.

[1473] Step 11:

[1474] The generation algorithm extracts and analyzes the photo's metadata (such as the date and time the photo was taken and GPS information).

[1475] Step 12:

[1476] The server stores the extracted metadata in a database and organizes the parsed information.

[1477] Step 13:

[1478] The server uses an interactive algorithm to generate questions for the user based on the parsed metadata.

[1479] Step 14:

[1480] The interactive algorithm generates questions and sends them to the user's device.

[1481] Step 15:

[1482] The user uses the device to answer questions posed by the conversational algorithm or to skip questions.

[1483] Step 16:

[1484] The device sends the user's answers to the server, which stores them in a database.

[1485] Step 17:

[1486] The server selects an appropriate travelogue template based on the information it collects (metadata and user responses).

[1487] Step 18:

[1488] The server fills in the information in the selected template and automatically generates a detailed travelogue.

[1489] Step 19:

[1490] The server provides the generated travelogue to the user's terminal, and the user views the travelogue.

[1491] Step 20:

[1492] Users can use their devices to set their travel journals to be public or private.

[1493] Step 21:

[1494] The terminal sends the configuration information to the server, which stores it in a database.

[1495] Step 22:

[1496] Other users can use their devices to view your published travelogue.

[1497] Step 23:

[1498] The server generates a virtual travel experience based on the published travelogue and provides it to other users.

[1499] Step 24:

[1500] The server also provides detailed information about related places and activities to support virtual travel.

[1501] These steps allow users to easily organize their travel photos, automatically generate and share detailed travel journals, and share their travel journals with other users to enjoy virtual travel experiences.

[1502] Example 1

[1503] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1504] Efficiently organizing the many images taken during a trip and generating a detailed travelogue based on them is a time-consuming and labor-intensive process. There is also a need for a method to share travelogues with other users and provide them with virtual travel experiences. However, existing systems lack the functionality to allow users to easily analyze image metadata and automatically generate detailed travelogues. Furthermore, they lack an interactive algorithm that generates questions based on user answers, and an effective means to provide virtual travel experiences.

[1505] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1506] In this invention, the server includes a means for users to upload multiple images via their terminals, a means for the server to store the images in cloud storage, a means for a generation algorithm to extract and analyze metadata from the images, a means for an interactive algorithm to ask the user detailed information and store the answers, a means for automatically generating a detailed travelogue based on the generation algorithm and the user's answers, and a means for enabling the automatically generated travelogue to be shared among users. This allows users to easily upload images taken during their trip and automatically generate and share a detailed travelogue based on the images. A virtual travel experience is also provided, allowing other users to share their travelogues and enjoy virtual experiences.

[1507] "User" means the person who uses the system to upload images or input information.

[1508] A "terminal" is an electronic device such as a computer or smartphone used by a user.

[1509] "Images" are photo files taken by users during their travels and uploaded to the system.

[1510] The "server" refers to a central management system that stores images and analyzes metadata.

[1511] "Cloud storage" is an external service for storing and managing data via the Internet.

[1512] "Generation algorithm" refers to software that extracts and analyzes image metadata.

[1513] "Metadata" is additional information including the date and time an image was taken and GPS information.

[1514] An "interactive algorithm" is software that generates questions to gather more information from users and stores their answers.

[1515] A "travel journal" is a written record generated based on images and information provided by the user.

[1516] "Virtual Travel Experience" is a feature that allows other users to virtually experience travel based on automatically generated travelogues.

[1517] This invention relates to a system that allows a user to upload multiple images taken during a trip, analyzes the date, time, location, and detailed information from those images, and automatically generates a travelogue. Specific embodiments for implementing this system are described below.

[1518] User Registration and Login

[1519] A user accesses the system using a terminal and first registers. The user enters their name, email address, and password into a registration form, and the terminal sends this information to the server. The server receives the entered information and creates a new user account in the database. When the user logs in, they enter their email address and password and send them to the server. The server refers to the database to authenticate them, and if successful, starts a session.

[1520] Specific behavior:

[1521] The user fills in the required information in the registration form and clicks the "Register" button.

[1522] The device sends the input to the server.

[1523] The server creates a new user account in the database.

[1524] Photo upload

[1525] The user uploads multiple photos taken during their trip from their device. The device sends a request to upload the selected photos to the server, and the server stores the received photos in cloud storage such as Google Cloud Storage. The server records the storage location of each photo in a database.

[1526] Specific behavior:

[1527] The user selects multiple photos from a file folder and clicks the "Upload" button.

[1528] The terminal transmits the selected photo file to the server.

[1529] The server stores the photo in Google Cloud Storage and records the URL in a database.

[1530] Photo analysis

[1531] The server sequentially retrieves photos stored in cloud storage and extracts the photo metadata (date and time of shooting, GPS information, etc.) using image analysis libraries such as OpenCV and TensorFlow. The extracted metadata is then stored in a database by the server.

[1532] Specific behavior:

[1533] The server retrieves the photos from the cloud storage.

[1534] The server uses an image analysis library to extract the metadata.

[1535] The extracted metadata is stored in a database.

[1536] Gathering more information with conversational AI

[1537] The server uses an interactive algorithm to generate questions based on the parsed metadata, sends the generated questions to the user's device, and the user can answer or skip the questions. The device then sends the user's answers to the server, which stores them in a database.

[1538] Specific behavior:

[1539] The server generates questions based on the metadata.

[1540] Example questions: "Where was this photo taken?"

[1541] The device displays a question to the user.

[1542] The user enters the answer and the device sends the answer to the server.

[1543] Automatic travel report generation

[1544] The server selects an appropriate travelogue template based on the collected information (analysis metadata and user responses), and automatically generates a detailed travelogue using a generative AI model such as GPT-3. The generated travelogue is then sent to the user's device for viewing.

[1545] Example prompt sentence:

[1546] "These are travelogues to Tokyo. Please generate a detailed travelogue based on the information below."

[1547] Specific behavior:

[1548] The server sends a prompt to the generative AI model.

[1549] The generated travelogue is sent to the user's device.

[1550] Information sharing and virtual travel experiences

[1551] Users set their travel journals to be public or private and send the setting information to the server. Travel journals that are public can be viewed by other users. The server generates a virtual travel experience based on the published travel journal and provides it to other users.

[1552] Specific behavior:

[1553] Users can set their travel journals to be public or private.

[1554] The server acquires publicly set travelogues and generates a virtual travel experience.

[1555] By implementing this system, users can easily organize their travel photos, automatically generate detailed and attractive travel journals, and share them with other users to enjoy virtual travel experiences.

[1556] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1557] Step 1: User Registration

[1558] The user enters their name, email address, and password in plain text.

[1559] The device sends this input information to the server in JSON format.

[1560] The server stores the received information in its database as a new user account.

[1561] Enter your name, email address, and password.

[1562] Data processing: Convert input information into JSON format.

[1563] Output: A new user account is created in the database.

[1564] Specific behavior:

[1565] The user enters "Yamada Taro", "taro@example.com", and a password in the registration form and clicks the "Register" button.

[1566] The device sends this information to the server in JSON format.

[1567] The server receives the information and creates a new user account in the database.

[1568] Step 2: Log in

[1569] The user enters their email address and password in plain text.

[1570] The device sends this input information to the server in JSON format.

[1571] The server refers to the database and performs authentication. If authentication is successful, a session is started.

[1572] Enter your email address and password.

[1573] Data calculation: Matching input information with existing information in a database.

[1574] Output: A session ID is generated.

[1575] Specific behavior:

[1576] The user enters "taro@example.com" and their password in the login form and clicks the "Login" button.

[1577] The device sends this information to the server in JSON format.

[1578] The server queries the database, and since authentication is successful, generates a session ID.

[1579] Step 3: Upload your photos

[1580] The user selects multiple photos taken during their trip from their device. The selection is made in file format.

[1581] The device sends the selected photo file to the server via an HTTP POST request.

[1582] The server stores the received photos in Google Cloud Storage and records their URLs in the database.

[1583] Input: Travel photo files.

[1584] Data processing: Convert the photo file into HTTP POST request format.

[1585] Output: The photo is saved to Google Cloud Storage and its URL is recorded in the database.

[1586] Specific behavior:

[1587] The user selects "IMG_001.jpg" to "IMG_010.jpg" from the file folder and clicks the "Upload" button.

[1588] The device sends the selected photo file to the server via an HTTP POST request.

[1589] The server stores the received photos in Google Cloud Storage and records their URLs in the database.

[1590] Step 4: Photo analysis

[1591] The server retrieves photos sequentially from Google Cloud Storage.

[1592] The server uses image analysis libraries such as OpenCV and TensorFlow to extract photo metadata (such as the date and time the photo was taken, GPS information, etc.).

[1593] The server stores the extracted metadata in a database.

[1594] Input: Google Cloud Storage photo URL.

[1595] Data Computation: Extraction of metadata using image analysis libraries.

[1596] Output: The extracted metadata is stored in a database.

[1597] Specific behavior:

[1598] The server retrieves "IMG_001.jpg" from Google Cloud Storage.

[1599] The server analyzes the EXIF ​​information of the photo using OpenCV and extracts the date and time of the photo and GPS information.

[1600] The extracted metadata is stored in a database.

[1601] Step 5: Gathering more information with conversational AI

[1602] The server generates questions based on the parsed metadata, using a generative AI model (e.g., GPT-3).

[1603] The device displays the generated question to the user, who can then enter an answer or skip.

[1604] The device sends the user's answers to the server, which stores them in a database.

[1605] Input: Metadata.

[1606] Data Computing: Generate questions and store answers using generative AI models.

[1607] Output: The generated questions and the user's answers are stored in a database.

[1608] Specific behavior:

[1609] Based on the metadata, the server sends the question "Where was this photo taken?" as a prompt to the generative AI model and receives the question as an answer.

[1610] The device displays a question to the user.

[1611] The user answers "Tokyo Tower," and the device sends the answer to the server.

[1612] The server stores the answer in a database.

[1613] Step 6: Automatic travel journal generation

[1614] The server selects an appropriate travelogue template based on the collected metadata and the user's answers.

[1615] The server automatically generates a detailed travelogue using a generative AI model (e.g., GPT-3).

[1616] The server sends the generated travelogue to the user's terminal.

[1617] Input: Metadata, user answers.

[1618] Data computation: Generating travel journals using generative AI models.

[1619] Output: The automatically generated travelogue is sent to the user's device.

[1620] Specific behavior:

[1621] The server converts the metadata and answers into prompts and sends them to the generative AI model.

[1622] The generated travelogue is sent to the user's device and can be viewed by the user.

[1623] Step 7: Information sharing and virtual travel experiences

[1624] The user sets the travel journal to be public or private and sends it from the device to the server.

[1625] The server stores the configuration information in a database.

[1626] The server generates a virtual travel experience based on the published travelogue and provides it to other users.

[1627] Input: Public / Private settings.

[1628] Data calculation: Converting published travel journal data into virtual experience content.

[1629] Output: The publicly set travelogue is shared with other users, providing them with a virtual travel experience.

[1630] Specific behavior:

[1631] The user sets whether their travel journal is public or private, and sends the setting information from their device to the server.

[1632] The server stores the configuration information in a database.

[1633] The server acquires the published travelogue data, generates a virtual travel experience, and provides it to other users.

[1634] The above is the specific processing flow of this system and the operation at each step.

[1635] (Application example 1)

[1636] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1637] In today's world, many users find it time-consuming to simply organize photos taken during a trip and record their experiences in a detailed travel journal. It's also difficult to share those travel journals with others or provide virtual travel experiences. This often results in valuable travel experiences remaining buried. The present invention aims to solve these problems by providing a system that automatically and efficiently generates travel journals based on photos taken by users and allows them to publish and share them in a virtual store.

[1638] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1639] In this invention, the server includes: a means for a user to upload multiple images via a terminal; a means for the server to store the images in cloud storage; a means for a generation algorithm to extract and analyze metadata from the images; a means for an interactive algorithm to ask the user detailed information and store the answers; a means for automatically generating detailed records based on the generation algorithm and the user's answers; a means for enabling the automatically generated records to be shared among users; a means for providing a virtual experience based on the collected information and the generated records; and a means for publishing the generated records in a virtual store. This allows users to efficiently organize their travel photos, automatically generate and publish attractive travel journals, and share and enjoy them with other users.

[1640] "User" means any person or entity that accesses the system, uploads multiple images, and inputs information to generate detailed records from those images.

[1641] "Terminal" means the hardware device used by a User to access the System and upload images or enter information.

[1642] "Server" refers to the back-end computing resources used to store images received from users and run generative and interactive algorithms.

[1643] "Cloud storage" refers to an external storage system that allows a server to safely store image data received via the Internet.

[1644] "Generation Algorithm" refers to the software process used to extract and analyze metadata from uploaded images and automatically generate detailed records.

[1645] "Metadata" refers to data that contains additional information specific to an image, such as the date and time the image was taken or GPS information.

[1646] "Interactive algorithm" refers to the software process by which a system generates questions to gather more information from users and stores their answers.

[1647] "Detailed Records" refers to travel journals and other documents that are automatically generated based on images and information provided by users.

[1648] "Virtual Store" refers to an online platform for publishing and sharing detailed user-generated records with other users.

[1649] A "virtual experience" is an experience that allows other users to take a virtual journey based on a publicly detailed record.

[1650] This invention relates to a system that allows users to upload multiple images taken during their trip, analyzes the date, time, location, and detailed information from those images, and automatically generates a detailed record. Specific embodiments for implementing this system and their program processing are described below.

[1651] User Registration and Login

[1652] A user accesses the system using a terminal and first registers. They enter information such as their name, email address, and password into the registration form, and the terminal sends this information to the server. The server receives the entered information and creates a new user account in its database. When logging in, the user enters their email address and password from the terminal and sends them to the server. The server refers to the database to perform authentication, and if successful, starts a session.

[1653] Photo upload

[1654] The user uploads multiple photos taken during their trip from their device. The device sends a request to upload the selected photos to the server, which then stores the photos in cloud storage. The server records the location of each photo in a database.

[1655] Photo analysis

[1656] The server sequentially passes photos stored in the cloud storage to a generation algorithm, which extracts and analyzes the photo's metadata (such as the date and time of the photo's capture, GPS information, etc.) The extracted metadata is then stored in a database by the server.

[1657] Gathering more information with conversational AI

[1658] The server uses an interactive algorithm to generate questions based on the parsed metadata. For example, questions like "Where was this photo taken?" or "What experience did you have here?" are sent to the user's device, and the user answers them. The user can also skip questions. The device sends the user's answers to the server, which stores them in a database.

[1659] Examples of specific prompts include:

[1660] 1. Question: "Where was this photo taken?"

[1661] User Answer: "This photo was taken at Kiyomizu-dera Temple in Kyoto."

[1662] 2. Question: "What was your experience here?"

[1663] User Answer: "I enjoyed visiting tourist spots and eating my way around."

[1664] Detailed automatic recording

[1665] The server selects an appropriate template based on the collected information (analysis metadata and user responses) and automatically generates a detailed record, which includes text, photos, anecdotes, etc. The generated record is then delivered to the user's device for viewing.

[1666] Virtual Experience and Information Sharing

[1667] Users can use their devices to set their detailed records to public or private. The setting information is sent to the server, which stores it in a database. Publicly set detailed records can be viewed by other users. The server generates a virtual experience based on the publicly set detailed records and provides it to other users, along with detailed information about related locations and activities.

[1668] The system is implemented using the following hardware and software:

[1669] Server: Cloud server, database (e.g., Amazon RDS)

[1670] Image storage: Cloud storage (e.g. AWS S3)

[1671] Generative AI models: Libraries for running generative algorithms, such as TensorFlow and PyTorch

[1672] This allows users to efficiently organize their travel photos, automatically generate and publish compelling travel journals, and share and enjoy them with other users.

[1673] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1674] Step 1: User registration and login

[1675] A user accesses the system using a terminal, enters their name, email address, and password, and submits it. The server receives this information and creates a new user account in the database. When logging in, the user enters their email address and password. The server uses this information to look up their database and authenticates them, and if successful, starts a session.

[1676] Input: User's name, email address, password

[1677] Output: New user account created and authenticated.

[1678] Step 2: Upload a photo

[1679] Users select multiple travel photos from their devices and send an upload request to the server, which then stores the photos in cloud storage and records their location in a database.

[1680] Input: Multiple photos taken by the user

[1681] Output: Save photos to cloud storage and record their location in a database

[1682] Step 3: Photo analysis

[1683] The server passes the photos stored in the cloud storage to a generation algorithm, which extracts and analyzes the photo's metadata (date and time of the photo, GPS information), which is then stored in a database by the server.

[1684] Input: Photos stored in cloud storage

[1685] Output: Extracted metadata stored in a database

[1686] Step 4: Gathering more information with conversational AI

[1687] The server uses an interactive algorithm to generate questions based on the parsed metadata. The generated questions are sent to the user's device, and the user answers them. The device then sends the user's answers to the server, which stores them in a database.

[1688] Input: extracted metadata, user answers

[1689] Output: Generated questions, saving user answers to database

[1690] Examples of specific prompts include:

[1691] 1. Question: "Where was this photo taken?"

[1692] User Answer: "This photo was taken at Kiyomizu-dera Temple in Kyoto."

[1693] 2. Question: "What was your experience here?"

[1694] User Answer: "I enjoyed visiting tourist spots and eating my way around."

[1695] Step 5: Automatically generate detailed records

[1696] The server automatically generates a detailed record based on the generation algorithm and the user's answers. The generated record includes text, photos, anecdotes, etc. The server then provides the generated record to the user's device.

[1697] Input: Analysis metadata, user answers

[1698] Output: Auto-generated detailed transcript

[1699] Step 6: Virtual Experience and Information Sharing

[1700] Users can use their devices to set their detailed records to public or private. The server stores this setting information in a database. Publicly set detailed records can be viewed by other users. The server generates a virtual experience based on the publicly set detailed records and provides it to other users.

[1701] Input: User's public / private settings

[1702] Output: Publishing detailed records and generating virtual experiences

[1703] These processing flows allow users to efficiently organize photos taken during their trip, automatically generate and publish detailed and engaging travel journals, and share their virtual experiences with other users.

[1704] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1705] The present invention relates to a system that allows users to upload multiple photos taken during their trip, analyzes the date, time, location, and detailed information from those photos, and automatically generates a travel diary. Furthermore, by combining this system with an emotion engine, the system can recognize the user's emotions and reflect them in the travel diary. Specific embodiments of the present invention and the processing of the program are described below.

[1706] User Registration and Login

[1707] A user accesses the system using a terminal and enters the required information such as name, email address, password, etc. into a new registration form. The terminal sends this information to the server, which receives it and creates a new user account in the database.

[1708] When logging in, the user enters their email address and password from their device and sends them to the server. The server then refers to the database for authentication, and if authentication is successful, a session begins.

[1709] Photo upload

[1710] The user selects multiple photos taken during their trip using their device and makes an upload request. The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[1711] Photo analysis

[1712] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts metadata from the photos and analyzes information such as date, time, and location. The extracted metadata is then stored in a database by the server.

[1713] Gathering more information with conversational AI

[1714] The server uses an interactive algorithm to generate questions for the user based on the parsed metadata. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. The user can also skip questions. The device sends the answers to the server, which stores them in a database.

[1715] Emotional regulation by emotion engine

[1716] The emotion engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, when a user inputs an emotion expression such as "I had fun" or "I'm a little tired," the emotion engine analyzes the information and generates emotion data. The emotion data is then sent to the server and stored in a database.

[1717] Automatic travel report generation

[1718] The server selects an appropriate travelogue template based on the collected metadata, user responses, and emotion data. A detailed travelogue is automatically generated based on the selected template. This travelogue includes text, photos, anecdotes, and the user's emotions. The generated travelogue is then provided to the user's device for viewing.

[1719] Information sharing and virtual travel experiences

[1720] Users can set their travel journals to be public or private using their devices. The setting information is sent to the server and stored in a database. Travel journals set to public can be viewed by other users. The server generates a virtual travel experience based on the published travel journal and provides other users with detailed information on related places and activities.

[1721] For example, if a user uploads photos taken during a trip and inputs the emotion "It was fun" through the emotion engine, the system will analyze this information and reflect it in the travelogue as a "fun memory." In addition, when other users view this travelogue, content will be provided that allows them to feel a "fun atmosphere" as if they are traveling virtually.

[1722] This allows the system of the present invention to not only allow users to easily organize their travel photos and automatically generate detailed travel journals, but also to create records that reflect the user's emotions and share them with other users. Furthermore, other users can gain information and experiences about new travel destinations through virtual travel experiences.

[1723] The processing flow will be explained below.

[1724] Step 1:

[1725] A user accesses the system using a terminal and enters the required information, such as name, email address, and password, into the new registration form.

[1726] Step 2:

[1727] The terminal sends the entered user information to the server, which then creates a new user account in the database.

[1728] Step 3:

[1729] The user uses a device to enter their email address and password into the login form to request a login.

[1730] Step 4:

[1731] The terminal sends authentication information to the server, which then refers to the database to authenticate the user. If authentication is successful, the server starts a session for the user.

[1732] Step 5:

[1733] A user uses a device to select multiple travel photos and make an upload request.

[1734] Step 6:

[1735] The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[1736] Step 7:

[1737] The server sequentially passes the photos stored in cloud storage to the generation algorithm.

[1738] Step 8:

[1739] The generation algorithm extracts and analyzes the metadata of the photo (such as the date and time of the photo being taken, GPS information, etc.), and the extracted metadata is stored in a database by the server.

[1740] Step 9:

[1741] The server uses an interactive algorithm to generate questions for the user based on the parsed metadata.

[1742] Step 10:

[1743] The conversational algorithm generates questions and sends them to the user's device, such as "Where was this photo taken?" or "What kind of experience did you have here?"

[1744] Step 11:

[1745] The user uses the device to answer questions posed by the conversational algorithm or to skip questions.

[1746] Step 12:

[1747] The device sends the user's answers to the server, which stores them in a database.

[1748] Step 13:

[1749] The conversational algorithm uses an emotion engine to recognize the user's emotions. For example, if the user answers "I had fun" or "I'm a little tired," the emotion will be recognized.

[1750] Step 14:

[1751] The emotion engine generates the recognized emotion data and sends it to the server, which stores the emotion data in a database.

[1752] Step 15:

[1753] The server selects an appropriate travel diary template based on the collected metadata, user responses, and sentiment data.

[1754] Step 16:

[1755] The server automatically generates a detailed travelogue by filling in the selected template with information, including text, photos, anecdotes, and the user's emotions.

[1756] Step 17:

[1757] The server provides the generated travelogue to the user's terminal, and the user views the travelogue.

[1758] Step 18:

[1759] Users can use their devices to set their travel journals to be public or private.

[1760] Step 19:

[1761] The terminal sends the configuration information to the server, which stores the configuration information in a database.

[1762] Step 20:

[1763] Other users can use their devices to view your published travelogue.

[1764] Step 21:

[1765] The server generates a virtual travel experience based on the published travelogue. Emotional data is also taken into consideration, and content adapted to the user's emotions is provided in the virtual travel experience.

[1766] Step 22:

[1767] The server provides detailed information about places and activities related to the virtual travel experience, and helps other users gain information and experiences about new travel destinations through the travelogue.

[1768] These steps allow users to easily organize their travel photos, automatically generate and share detailed and emotional travel journals, and allow other users to experience virtual travel through the journals.

[1769] Example 2

[1770] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1771] Conventional methods require users to organize multiple photos taken during a trip and automatically generate a travelogue by analyzing the date, time, location, and detailed information from those photos. This requires a lot of effort and effort. It is also difficult to create a travelogue that reflects emotions, and it is not easy to share or provide a virtual travel experience. There is a need for a system that solves these problems and enables efficient automatic generation and sharing of travelogues that reflect emotions.

[1772] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1773] In this invention, the server includes means for a user to upload multiple images via a device, means for the server to store the images in a cloud storage device, means for a generation algorithm to extract and analyze metadata for the images, means for an interactive algorithm to ask the user detailed information and store the responses, means for an emotion recognition engine to recognize the user's emotions and analyze and store the emotion data, means for automatically generating detailed travelogues based on the generation algorithm, the user's responses, and the emotion data, and means for enabling the automatically generated travelogues to be shared among users, thereby enabling users to efficiently organize photos taken during their trips, automatically generate detailed travelogues, and create travelogues that reflect their emotions and share them with other users.

[1774] "User" refers to any individual or entity that uses the System to generate travel journals and upload photos.

[1775] "Device" refers to the hardware used by a user, such as a computer, smartphone, or tablet.

[1776] "Images" refers to photographic data taken by users during their travels and uploaded to the system.

[1777] "Server" refers to a computer system that controls the entire system and exchanges data with users.

[1778] "Cloud storage" refers to a storage service for storing data via the Internet.

[1779] "Generation algorithm" refers to a program for extracting and analyzing image metadata.

[1780] "Metadata" refers to additional information contained in an image, such as the date, time, location, and shooting conditions.

[1781] An "interactive algorithm" is a program designed to ask users detailed questions and collect their answers.

[1782] "Emotion recognition engine" refers to a system for analyzing and storing user emotional data.

[1783] "Emotional data" refers to information analyzed by an emotion recognition engine based on emotional expressions entered by the user.

[1784] "Records" refers to detailed travel journals that are automatically generated based on generative algorithms, user responses, and sentiment data.

[1785] "Virtual travel experience" refers to content that allows other users to experience the atmosphere and information of a trip based on publicly available records.

[1786] "Publicity Settings" refers to the settings options that allow a User to make a Recording that the User generates available to other Users.

[1787] The present invention relates to a system that allows users to upload multiple photos taken during their trip, analyzes the date, time, location, and detailed information from those photos, and automatically generates a travel diary. Furthermore, by combining this system with an emotion recognition engine, the system can recognize the user's emotions and reflect them in the travel diary. Specific embodiments of the present invention and the processing of the program will be described below.

[1788] User Registration and Login

[1789] A user accesses the system using a terminal and enters the required information (such as name, email address, and password) in a new registration form. The terminal sends this information to the server, which then stores it in a database and creates a new user account. When logging in, the user enters their email address and password from the terminal and sends them to the server, which then references the database to perform authentication and starts a session if authentication is successful.

[1790] Photo upload

[1791] A user uses a device to select multiple photos taken during a trip and request uploading. The device sends the selected photo data to a server, which then stores the received photos in a cloud storage device and records the storage location of each photo in a database.

[1792] Photo analysis

[1793] The server processes the photos stored in the cloud storage device sequentially. Specifically, it uses a generative algorithm (e.g., Google Cloud Vision API) to extract and analyze the photo metadata (date, time, location, etc.). The server stores the analyzed metadata in a database.

[1794] Gathering more information with conversational AI

[1795] Based on the analyzed metadata, the server uses a conversational algorithm (e.g., GPT-3) to generate questions for the user. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. Prompt sentences such as "Where was this photo taken?" or "How did you feel when you took the photo?" may also be used. The device sends the answers to the server, which stores them in a database. The user can also skip questions.

[1796] Emotional regulation by emotion engine

[1797] The emotion recognition engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, when a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion recognition engine analyzes it and generates emotional data. The emotional data is then sent to the server and stored in a database.

[1798] Automatic travel report generation

[1799] The server selects an appropriate travelogue template based on the collected metadata, the user's responses, and sentiment data. For example, it generates a detailed travelogue based on information such as "I took the photo at the top of the Eiffel Tower. The view was amazing." The generated travelogue is then sent to the user's device for viewing.

[1800] Information sharing and virtual travel experiences

[1801] The user sets the travel journal to be public or private using the device. This setting information is sent to the server and stored in a database. The server makes publicly set travel journals available to other users. The server also generates a virtual travel experience based on the published travel journal and provides other users with detailed information on related places and activities.

[1802] In this way, the system of the present invention allows users to efficiently organize their travel photos, automatically generate detailed travel journals, create emotional records, and share them with other users, allowing them to gain information and experiences about new travel destinations.

[1803] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1804] Step 1: User registration and login

[1805] A user accesses the system using a terminal and enters the required information (name, email address, password, etc.) into a new registration form. This becomes input data. The terminal sends this information to the server. The server stores the received information in a database and creates a new user account. This becomes output data. When logging in, the email address and password entered by the user are sent from the terminal to the server. The server refers to the database to perform authentication, and if authentication is successful, a session is started. This allows the user to access the system.

[1806] Step 2: Upload a photo

[1807] A user uses a device to select multiple photos taken during a trip and make an upload request. This is the input data. The device sends the selected photo data to the server. The server stores the received photos in cloud storage and records the storage location of each photo in a database. This is the output data. The photos are then stored in the system.

[1808] Step 3: Photo analysis

[1809] The server sequentially passes the photos stored in the cloud storage device to a generation algorithm (e.g., Google Cloud Vision API). This becomes the input data. The generation algorithm extracts and analyzes the photo's metadata (date, time, location, etc.). The server stores the analyzed metadata in a database. This becomes the output data. As a result, detailed information about the photo is extracted and stored in the database.

[1810] Step 4: Gathering more information with conversational AI

[1811] Based on the analyzed metadata, the server uses an interactive algorithm (e.g., GPT-3) to generate questions for the user. This becomes the input data. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. This becomes the output data. The server sends the generated questions to the user's device, and the user answers the questions. The device sends the answers to the server, and the server stores them in a database. The user can also skip questions, allowing more information to be collected.

[1812] Step 5: Emotional regulation with the Emotion Engine

[1813] The emotion recognition engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. This becomes the input data. For example, when a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion recognition engine analyzes it and generates emotional data. The emotional data is sent to the server and stored in a database. This becomes the output data. This records the user's emotions.

[1814] Step 6: Automatic travel journal generation

[1815] The server selects an appropriate travelogue template based on the collected metadata, user responses, and emotional data. This becomes the input data. A detailed travelogue is generated by incorporating the metadata, episodes, and emotional information into the template. The generated travelogue is then provided to the user's device. This becomes the output data. The user can then view the detailed travelogue.

[1816] Step 7: Information sharing and virtual travel experiences

[1817] The user uses their device to set their travel journal to be public or private. This becomes the input data. The setting information is sent to the server and stored in a database. The server then sets the public travel journal so that other users can view it. This becomes the output data. The server also generates virtual travel experience content based on the published travel journal and provides other users with detailed information about related places and activities. This allows other users to gain information and experiences about new travel destinations.

[1818] (Application example 2)

[1819] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1820] Conventional travel journal creation systems require users to manually organize photos and detailed information to create a travel journal, a time-consuming and labor-intensive process. It is also difficult to reflect the user's emotions, making it difficult to gain emotional empathy when sharing with other users. Furthermore, automatic travel journal generation in video format is limited, making it difficult to provide an effective visual experience.

[1821] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1822] In this invention, the server includes a means for a user to upload multiple images via a device, a means for the server to store the images in cloud storage, a means for a generation algorithm to extract and analyze metadata of the images, a means for an interactive algorithm to ask the user detailed information and store the answers, a means for analyzing the user's emotions in real time using an emotion engine to generate emotion data, a means for automatically generating detailed video recordings based on the generation algorithm, the user's answers, and the emotion data, and a means for enabling the automatically generated recordings to be shared among users. This allows users to easily upload photos, automatically generate emotionally-reflecting travel journals in video format, and share them with other users.

[1823] "User" means a person who uses a digital device to access the system and create and share travel journals.

[1824] "Device" refers to electronic devices with communication capabilities, such as computers, smartphones, and tablets.

[1825] "Images" refers to visual data such as photographs and illustrations taken by users during their travels.

[1826] A "server" is a computer system within a network that receives, processes, and stores data submitted by users.

[1827] "Cloud storage" refers to a service that stores data on remote servers accessible via the Internet.

[1828] A "generative algorithm" is a computational method for automatically generating new content based on input data.

[1829] "Metadata" refers to additional information that accompanies an image, such as the date, time, location, and camera settings.

[1830] An "interactive algorithm" is a program that collects information through dialogue with the user and generates appropriate questions.

[1831] An "emotion engine" is a program that analyzes emotions from user input and actions and generates them as data.

[1832] "Emotion data" is data that indicates the user's emotional state, obtained as a result of analysis by the emotion engine.

[1833] "Video recording" refers to content that integrates images, audio, and text for dynamic playback.

[1834] "Automatically Generated Records" refers to documents or media content that are automatically generated based on data collected by the system.

[1835] "Shareable" means that the records created are viewable by other users or on public platforms, allowing for the widespread dissemination of information.

[1836] A "virtual travel experience" refers to a digital simulation that allows users to experience a place through sight, sound, and other senses without actually visiting the place.

[1837] The present invention provides a system for automatically generating travel video that reflects emotions based on travel photos taken by a user, and sharing the video with other users. Specific embodiments are described below.

[1838] User Registration and Login

[1839] First, a user accesses the system using a device (smartphone or PC) and enters the required information such as name, email address, and password into the new registration form. This information is sent to the server, which then creates a new user account in the database. When logging in, the user enters their email address and password from their device and sends them to the server. The server then refers to the database to perform authentication, and if authentication is successful, a session begins.

[1840] Photo upload

[1841] The user selects multiple photos taken during their trip from their device and makes an upload request. The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[1842] Photo analysis

[1843] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts metadata from the photos and analyzes information such as date, time, and location. The extracted metadata is then stored in a database by the server.

[1844] Gathering more information with conversational AI

[1845] The server uses an interactive algorithm based on the analyzed metadata to generate questions for the user. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. The user can also skip questions. The user's answers are sent from the device to the server, which stores them in a database.

[1846] Emotional regulation by emotion engine

[1847] The emotion engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, if a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion engine analyzes that information and generates emotional data. The emotional data is then sent to a server and stored in a database.

[1848] Automatic travel report generation

[1849] The server selects an appropriate video recording template based on the collected metadata, user responses, and emotion data. Based on the selected template, a detailed travel video is automatically generated. This video recording includes text, photos, anecdotes, and the user's emotions. The generated video recording is then provided to the user's device for viewing.

[1850] Information sharing and virtual travel experiences

[1851] Users can use their devices to set the generated travelogue video as public or private. The setting information is sent to the server and stored in a database. Travelogue videos set as public can be viewed by other users. The server generates a virtual travel experience based on the published travelogue and provides other users with detailed information about related places and activities.

[1852] Specific examples

[1853] For example, suppose a user uploads a photo of Tokyo Tower taken during a trip. The system analyzes the photo's metadata and automatically obtains "Tokyo Tower information." The user then inputs "I had fun at this place" into the emotion engine. Based on this information, a video travelogue is generated as a fun memory.

[1854] Example prompts for generative AI models

[1855] Upload a photo of Tokyo Tower and write a travel journal about the fun you had.

[1856] In this way, the system of the present invention allows users to easily organize travel photos, automatically create travel journals that reflect their emotions, and share them with other users.

[1857] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1858] Step 1: User registration and login

[1859] First, a user accesses the system using a terminal and enters the required information such as name, email address, and password into a new registration form. The terminal sends this information to the server, which receives the information and creates a new user account in the database. When logging in, the user enters their email address and password from the terminal and sends them to the server. The server refers to the database to perform authentication, and if authentication is successful, a session begins.

[1860] Input: User information (name, email address, password)

[1861] Output: User account creation and authentication session

[1862] Step 2: Upload a photo

[1863] The user selects multiple photos taken during their trip from their device and makes an upload request. The device sends the selected photo data to the server, which then stores the photos in cloud storage. The server records the storage location of each photo in a database.

[1864] Input: A photo selected by the user

[1865] Output: Photos saved in cloud storage and a record of their location

[1866] Step 3: Photo analysis

[1867] The server sequentially passes photos stored in cloud storage to a generation algorithm, which extracts metadata from the photos and analyzes information such as date, time, and location. The extracted metadata is then stored in a database by the server.

[1868] Input: Photos stored in cloud storage

[1869] Output: Extracted and parsed metadata (date, time, location, etc.)

[1870] Step 4: Gathering more information with conversational AI

[1871] The server uses an interactive algorithm based on the analyzed metadata to generate questions for the user. For example, questions such as "Where was this photo taken?" or "What experience did you have here?" are generated. These questions are sent to the user's device, and the user answers them. The user can also skip questions. The user's answers are sent from the device to the server, which stores them in a database.

[1872] Input: Parsed metadata

[1873] Output: User details and answer questions

[1874] Step 5: Emotional regulation with the Emotion Engine

[1875] The emotion engine works in conjunction with the conversational algorithm to recognize the user's emotions in real time. For example, if a user inputs an emotional expression such as "I had fun" or "I'm a little tired," the emotion engine analyzes that information and generates emotional data. The emotional data is then sent to a server and stored in a database.

[1876] Input: User's emotional expression

[1877] Output: Generated emotion data

[1878] Step 6: Automatic travel journal generation

[1879] The server selects an appropriate video recording template based on the collected metadata, user responses, and emotion data. Based on the selected template, a detailed travel video is automatically generated. This video recording includes text, photos, anecdotes, and the user's emotions. The generated video recording is then provided to the user's device.

[1880] Input: Metadata, user responses, sentiment data

[1881] Output: Automatically generated travel video

[1882] Step 7: Information sharing and virtual travel experiences

[1883] Users can use their devices to set the generated travelogue video as public or private. The setting information is sent to the server and stored in a database. Travelogue videos set as public can be viewed by other users. The server generates a virtual travel experience based on the published travelogue and provides other users with detailed information about related places and activities.

[1884] Input: Information set to public or private

[1885] Output: Published travelogues and virtual travel experiences

[1886] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1887] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1888] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1889] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1890] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1891] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1892] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1893] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1894] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1895] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1896] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1897] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1898] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1899] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1900] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1901] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1902] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1903] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1904] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1905] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1906] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1907] The following is further disclosed regarding the above embodiment.

[1908] (Claim 1)

[1909] A means for users to upload multiple images via their device;

[1910] A server stores the image in a cloud storage;

[1911] means for a generation algorithm to extract and analyze metadata for said images;

[1912] A means for interactive algorithms to ask users for more information and store their answers;

[1913] means for automatically generating a detailed record based on said generation algorithm and the user's responses;

[1914] The system includes means for enabling the automatically generated records to be shared among users.

[1915] (Claim 2)

[1916] The system of claim 1 , further comprising means for enabling a virtual travel experience based on the published automatically generated recording to provide a virtual experience.

[1917] (Claim 3)

[1918] 10. The system of claim 1, further comprising means for allowing a user to skip a question when the interactive algorithm presents the question to the user.

[1919] "Example 1"

[1920] (Claim 1)

[1921] A means for a user to upload multiple images via a device;

[1922] A server stores the image in a cloud storage;

[1923] means for a generation algorithm to extract and analyze metadata for said images;

[1924] A means for interactive algorithms to ask users for more information and store their answers;

[1925] means for automatically generating a detailed travelogue based on the generation algorithm and the user's responses;

[1926] The system further includes a means for enabling the automatically generated travelogue to be shared among users.

[1927] (Claim 2)

[1928] The system of claim 1 , further comprising means for enabling a virtual travel experience based on the published automatically generated travelogue.

[1929] (Claim 3)

[1930] 10. The system of claim 1, further comprising means for allowing a user to skip a question when the interactive algorithm presents the question to the user.

[1931] "Application Example 1"

[1932] (Claim 1)

[1933] A...

Claims

1. A means for users to upload multiple images via their device; A server stores the image in a cloud storage; means for a generation algorithm to extract and analyze metadata for said images; A means for interactive algorithms to ask users for more information and store their answers; means for automatically generating a detailed record based on said generation algorithm and the user's responses; The system includes means for enabling the automatically generated records to be shared among users.

2. The system of claim 1 , further comprising means for enabling a virtual travel experience based on the automatically generated recordings that are made public to provide a virtual experience.

3. The system of claim 1 , further comprising means for allowing the user to skip questions when the interactive algorithm presents the questions to the user.

Images