System

Abstract

A system is provided.SOLUTION: A system comprising: means for automatically collecting information from a social networking service on the Internet; means for identifying tourist spots and stores from the collected information using image recognition technology; means for extracting the identified information as text data using generative artificial intelligence technology; means for statistically analyzing the extracted data and classifying the data according to categories; means for designing a travel plan based on the classified information and automatically generating a schedule in consideration of time and budget; and means for displaying the generated travel plan on a user's device and allowing the user to confirm and adjust the travel plan.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the spread of social networking services (SNS), users are increasingly getting travel ideas from various sources on the Internet. However, using this information to create optimal travel plans is extremely time-consuming and laborious. Furthermore, the information is fragmented and not provided in a well-organized format, making it difficult for users to plan their trips efficiently. Furthermore, even after creating a travel plan, managing schedules and budgets can be cumbersome, making it difficult to create a plan that maximizes the enjoyment of the experiences users desire. [Means for solving the problem]

[0005] The present invention is a system that automatically collects information from social networking services on the Internet, analyzes the information using image recognition technology and generative artificial intelligence technology, and provides users with optimal travel plans. Specifically, it includes the following means.

[0006] 1. Automatically collect information from social networking services on the Internet.

[0007] 2. A means of identifying tourist spots and stores from collected information using image recognition technology.

[0008] 3. A means of extracting identified information as text data using generative artificial intelligence technology.

[0009] 4. A means of statistically analyzing the extracted data and classifying them into categories.

[0010] 5. A means of designing travel plans based on classified information and automatically generating schedules that take time and budget into consideration.

[0011] 6. A means of displaying the generated itinerary on the user's device, allowing the user to review and adjust it.

[0012] 7. A means for automatically generating a travel plan based on the adjustment information selected by the user.

[0013] 8. A means of transmitting finalized travel plans to partner travel providers.

[0014] 9. A means to provide a process for users to purchase and book travel plans.

[0015] This allows users to easily create a customized and optimal travel plan with just a click, allowing them to enjoy their trip efficiently.

[0016] The "Internet" is a large-scale information and communications network that interconnects computer networks around the world.

[0017] A "social networking service" is an online service that enables users to share information and interact with other users.

[0018] "Means of automatically collecting information" refers to methods or technologies for automatically obtaining specific data from the Internet using a program.

[0019] "Image recognition technology" is a technology in which a computer analyzes an image and identifies the objects and patterns contained within it.

[0020] A "tourist destination" is a place or area where tourists visit and enjoy tourism activities.

[0021] A "store" is a commercial facility that provides goods and services.

[0022] "Generative artificial intelligence technology" is a technology that enables artificial intelligence to learn patterns from large amounts of data and generate new information and content.

[0023] "Means of extracting text data" refers to methods and techniques for extracting necessary text information from collected information.

[0024] "Methods for statistically analyzing data" are methods and techniques that use statistical techniques to find significant information or patterns in data.

[0025] A "categorization tool" is a method or technique for grouping collected data based on specific criteria.

[0026] "Means for designing travel plans" are methods and techniques for creating optimal travel courses and schedules based on the traveler's wishes.

[0027] "Means for automatically generating a schedule" refers to methods or techniques for automatically creating a detailed timetable based on a travel plan.

[0028] A "user device" is an electronic device, such as a computer, smartphone, or tablet, through which a user obtains information and interacts.

[0029] "Adjustment Information" refers to information for changes or modifications that a user makes to a travel plan.

[0030] "Affiliated travel agents" refers to travel companies and travel agencies that work in conjunction with the system to provide and book travel products.

[0031] The "travel purchase process" refers to the series of actions or processes a user takes to purchase a confirmed travel plan and complete a booking. [Brief explanation of the drawings]

[0032] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0033] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0034] First, the terms used in the following description will be explained.

[0035] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0036] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0037] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0038] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0039] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0040] [First embodiment]

[0041] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0042] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0043] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0044] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0045] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0046] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0047] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0048] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0049] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0050] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0051] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0052] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0053] The present invention is a system that automatically collects information from social networking services (SNS) on the Internet, analyzes the information using image recognition technology and generative artificial intelligence technology, and provides users with optimal travel plans. The following describes in detail the embodiments of the present invention.

[0054] System configuration

[0055] This system mainly consists of a server and a user's device. The server collects information from SNS and analyzes and classifies the data using image recognition and generative artificial intelligence technology. The user's device receives the information, displays the travel plan, and allows adjustments.

[0056] Collecting SNS information

[0057] The server automatically collects posts based on specific keywords and hashtags using the APIs of social media platforms. This includes images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.). For example, it collects information based on hashtags such as "Tokyo travel" and "gourmet."

[0058] Image recognition and text data extraction

[0059] The server applies image recognition technology to the collected images and videos to identify place names, tourist attractions, stores, etc. For example, it automatically extracts famous tourist spots (Tokyo Tower and Sensoji Temple) and restaurants. It also uses generative artificial intelligence technology to extract additional text data from post captions and comment sections.

[0060] Data classification and analysis

[0061] The server statistically analyzes the extracted data using image recognition and text analysis, categorizing it into categories, such as tourist attractions, restaurants, and events, and generates rankings for each category. These rankings are displayed based on weekly, monthly, and yearly metrics based on the collected data.

[0062] Travel plan design

[0063] When a user clicks on an image or piece of information that interests them, the server automatically generates a travel plan based on that information. The plan includes places to visit, places to eat, activities, etc. For example, if a user clicks on an image of "Tokyo Tower," a schedule is automatically created that takes into account nearby tourist attractions, restaurants, and transportation options.

[0064] Schedule and budget management

[0065] The server generates a detailed schedule (timetable and budget) based on the travel plan and displays it on the user's device. The user can check this schedule and make changes as needed. For example, a detailed plan could be provided, such as visiting Sensoji Temple in the morning, having lunch at a famous okonomiyaki restaurant in Asakusa, and then visiting Tokyo Tower.

[0066] Travel agency collaboration and booking

[0067] The server sends the finalized travel plan to the partner travel agency, allowing the user to purchase and reserve the plan. The user's device is redirected to the travel agency's reservation page, where payment procedures are completed. In this way, the user can enjoy the optimal travel plan without any hassle.

[0068] Specific examples

[0069] For example, a user searches for "Kyoto travel" on a major online social networking site and clicks on a photo that catches their eye. This photo may show a famous tourist spot in Kyoto (e.g., Kiyomizu-dera Temple). Based on this information, the server automatically generates a travel plan that includes Kiyomizu-dera Temple and provides a schedule that takes into account nearby tourist spots, restaurants, and transportation options. The user can also review this plan and select additional tourist spots (e.g., Kinkaku-ji Temple) if necessary. Finally, once the plan is finalized, the server sends this information to the travel agency, and the reservation process is completed.

[0070] As described above, the present invention provides a system that utilizes information from SNS to enable users to easily automatically generate optimal travel plans.

[0071] The processing flow will be explained below.

[0072] Step 1:

[0073] The server uses the API of a social networking service (SNS) to search for posts based on specific keywords or hashtags (e.g., "travel," "tourist destinations") The data collected through the search includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc.

[0074] Step 2:

[0075] The server stores the collected social media post data in a temporary database, allowing for efficient subsequent analysis. It also stores the metadata of each post for analysis.

[0076] Step 3:

[0077] The server analyzes the collected images using image recognition technology to identify key features (e.g., landmarks, tourist attractions, restaurants), leveraging computer vision techniques to recognize specific landmarks and objects.

[0078] Step 4:

[0079] The server uses generative artificial intelligence techniques to extract relevant text data (e.g., place names, store names, activity details) from post captions and comment sections, and analyzes the extracted text data using natural language processing (NLP) techniques to identify important keywords and phrases.

[0080] Step 5:

[0081] The server statistically analyzes the data extracted through image recognition and text analysis. Specifically, it calculates how frequently each feature appears and evaluates its popularity and relevance. Based on this, the data is classified into categories such as tourist attractions, restaurants, and events.

[0082] Step 6:

[0083] The server creates weekly, monthly, and yearly rankings based on statistically analyzed data. This allows users to easily understand popular spots and activities for each season. This ranking information is also used as reference when creating plans.

[0084] Step 7:

[0085] When a user clicks on an image or piece of information that interests them on the social networking site, the device sends that information to the server, which then collects information on related tourist spots and stores and automatically generates a travel plan based on the user's input.

[0086] Step 8:

[0087] Based on the information the user clicks, the server designs a travel plan that combines tourist spots, restaurants, activities, etc. The generated plan also includes transportation and accommodations, and the schedule is automatically generated taking into account time and budget.

[0088] Step 9:

[0089] The device displays the generated itinerary to the user, who can review it and make adjustments as needed, such as changing the order of the stops or selecting additional tourist attractions.

[0090] Step 10:

[0091] After the user adjusts the plan, the device sends the information back to the server, which then regenerates the plan based on the user's adjustments and sends the final plan to the device.

[0092] Step 11:

[0093] The server sends the confirmed travel plan to the partner travel agency's system via API, allowing the user to purchase and reserve the travel plan directly. The terminal is then redirected to the travel agency's reservation page, where the user can complete payment procedures.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] Traditional travel planning requires users to manually gather vast amounts of information and create plans, which is time-consuming and labor-intensive. Furthermore, fragmented information on the Internet makes it difficult to efficiently generate comprehensive travel plans. Furthermore, the process of coordinating travel plans and making reservations is cumbersome, placing a significant burden on users.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes means for automatically collecting information from social networking services on the Internet, means for identifying tourist attractions and stores from the collected information using image recognition technology, means for extracting the identified information as text data using generative artificial intelligence technology, means for statistically analyzing the extracted data and classifying it by category, means for designing a travel plan based on the classified information and automatically generating a schedule that takes time and budget into consideration, means for displaying the generated travel plan on the user's device so that the user can confirm and adjust it, means for automatically generating a new travel plan based on information selected by the user, and means for transmitting the confirmed travel plan to affiliated travel agencies and providing procedures for the user to reserve and purchase the travel plan. This allows users to efficiently generate, adjust, and book an optimal travel plan without any effort.

[0099] A "social networking service" is an online platform that allows users to exchange information and communicate with each other over the Internet.

[0100] "Image recognition technology" is a technology in which a machine analyzes the content of an image and identifies the objects and scenes contained within it.

[0101] "Generative AI technology" is an AI technology that generates natural language sentences, images, etc. based on input data.

[0102] "Text data" refers to data consisting of character information, including sentences and words.

[0103] "Statistical analysis" means analyzing the characteristics and trends of collected data using statistical methods.

[0104] A "travel plan" refers to a plan of the schedule, destinations, activities, etc. for a trip within a specific period of time.

[0105] A "schedule" is a time-based plan that shows the sequence of activities or events that should occur at a particular time.

[0106] "User Device" means a hardware device used by a User, such as a computer, smartphone, or tablet.

[0107] "Travel Agent" means a business or organization that provides travel services, sells travel products, and processes reservations.

[0108] "Reservation procedure" refers to the action of finalizing travel plans and reserving accommodation and transportation based on those plans.

[0109] The present invention is a system that automatically collects information from social networking services (SNS) on the Internet, analyzes the information using image recognition technology and generative artificial intelligence technology, and provides users with optimal travel plans. A detailed description of an embodiment of the present invention will be given below.

[0110] System configuration

[0111] This system mainly consists of a server and a user's device. The server collects information from SNS and analyzes and classifies the data using image recognition and generative artificial intelligence technology. The user's device receives the information, displays the travel plan, and allows adjustments.

[0112] Collecting SNS information

[0113] The server automatically collects posts based on specific keywords and hashtags using the APIs of social media platforms. This includes images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.). For example, it collects information based on hashtags such as "Tokyo travel" and "gourmet."

[0114] Image recognition and text data extraction

[0115] The server applies image recognition technology to the collected images and videos to identify place names, tourist attractions, stores, etc. For example, it automatically extracts famous tourist spots (e.g., Tokyo Tower and Sensoji Temple) and restaurants. It also uses generative artificial intelligence technology to extract additional text data from post captions and comment sections.

[0116] Data classification and analysis

[0117] The server statistically analyzes the extracted data using image recognition and text analysis, categorizing it into categories, such as tourist attractions, restaurants, and events, and generates rankings for each category. These rankings are displayed based on weekly, monthly, and yearly metrics based on the collected data.

[0118] Travel plan design

[0119] When a user clicks on an image or piece of information that interests them, the server automatically generates a travel plan based on that information. The plan includes places to visit, places to eat, activities, etc. For example, if a user clicks on an image of "Tokyo Tower," a schedule is automatically created that takes into account nearby tourist attractions, restaurants, and transportation options.

[0120] Schedule and budget management

[0121] The server generates a detailed schedule (timetable and budget) based on the travel plan and displays it on the user's device. The user can check this schedule and make changes as needed. For example, a detailed plan could be provided, such as visiting Sensoji Temple in the morning, having lunch at a famous okonomiyaki restaurant in Asakusa, and then visiting Tokyo Tower.

[0122] Travel agency collaboration and booking

[0123] The server sends the finalized travel plan to the partner travel agency, allowing the user to purchase and reserve the plan. The user's device is redirected to the travel agency's reservation page, where payment procedures are completed. In this way, the user can enjoy the optimal travel plan without any hassle.

[0124] Specific examples

[0125] For example, a user searches for "Kyoto travel" on a major online social networking site and clicks on a photo that catches their eye. This photo may show a famous tourist spot in Kyoto (e.g., Kiyomizu-dera Temple). Based on this information, the server automatically generates a travel plan that includes Kiyomizu-dera Temple and provides a schedule that takes into account nearby tourist spots, restaurants, and transportation options. The user can also review this plan and select additional tourist spots (e.g., Kinkaku-ji Temple) if necessary. Finally, once the plan is finalized, the server sends this information to the travel agency, and the reservation process is completed.

[0126] Prompt Sentence Examples

[0127] "A user searched for 'Kyoto travel' on a major online social networking site and clicked on a photo that caught their eye (Kiyomizu-dera Temple). Based on this information, please create a Kyoto trip plan that includes Kiyomizu-dera Temple. Please provide a detailed itinerary that takes into account places to visit, restaurants, recommended activities, and transportation options."

[0128] In this way, the present invention provides a system that utilizes information from SNS to enable users to easily automatically generate optimal travel plans.

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Step 1:

[0131] The server automatically collects social media information. It uses the API of social media platforms, such as the Twitter API, to collect posts based on specific keywords and hashtags (for example, "Tokyo travel" or "food"). The input is the keywords and hashtags specified by the user. The output is the collected images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.). Specifically, the server calls the API to retrieve posts in real time and stores them in a database.

[0132] Step 2:

[0133] The server performs image recognition on the collected images and videos. It uses the Google (registered trademark) Cloud Vision API to identify tourist attractions and stores contained in the images. The input is the collected images and videos. The output is information about the identified tourist attractions and stores (e.g., Tokyo Tower, Sensoji Temple). Specifically, the server calls the image recognition API, analyzes objects and text in the images, and stores the results in a database.

[0134] Step 3:

[0135] The server extracts text data using generative artificial intelligence technology. It uses a generative AI model, such as OpenAI's GPT-3.5, to generate text data from the collected post captions and comments. The input is the post captions and comments. The output is the extracted text data. Specifically, the server sends prompts to the generative AI model, analyzes the generated text, and extracts important information.

[0136] Step 4:

[0137] The server statistically analyzes the extracted data and classifies it into categories, such as tourist attractions, restaurants, and events, and generates rankings for each category. The input is information about identified tourist attractions and stores and the extracted text data. The output is the data classified by category and rankings. Specifically, the server retrieves data from the database, applies statistical analysis algorithms to classify the data, and calculates rankings.

[0138] Step 5:

[0139] When a user clicks on an image or piece of information that interests them, the server automatically generates a travel plan based on that information. The input is the images and information selected by the user. The output is an automatically generated travel plan (places to visit, places to eat, activities, etc.). Specifically, the server runs a travel plan generation algorithm based on the selected data to create an optimal schedule for each tourist spot and shop.

[0140] Step 6:

[0141] The server generates a detailed schedule (timetable and budget) based on the travel plan and displays it on the user's device. The input is the automatically generated travel plan. The output is the detailed schedule and budget. Specifically, the server analyzes the travel plan and applies a time and budget management algorithm to determine a specific schedule and sends it to the user's device.

[0142] Step 7:

[0143] The server sends the final confirmed travel plan to the partner travel agency, allowing the user to purchase and book the travel plan. The input is the confirmed travel plan. The output is a notification of completion of transmission to the travel agency and the progress of the reservation. Specifically, the server calls the partner agency's API to send the travel plan and manages the reservation and purchase procedures. On the user's device, the server is redirected to the reservation page and the payment procedure is carried out.

[0144] Through the above steps, this system utilizes information from SNS to enable users to easily automatically generate and book optimal travel plans.

[0145] (Application example 1)

[0146] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0147] Conventional travel planning systems require users to manually search for information on tourist spots, restaurants, and other attractions that interest them and then create a plan based on that information, which is both time-consuming and laborious. Furthermore, generating an optimal travel plan requires integrating a large amount of information, which places a significant burden on users. Furthermore, reservations and payment procedures must be completed separately, which is time-consuming. To solve these issues, a system is needed that automatically collects information that interests users and provides optimal travel plans.

[0148] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0149] In this invention, the server includes means for automatically collecting information from social networking services on the Internet, means for identifying tourist attractions and stores from the collected information using image recognition technology, means for extracting the identified information as text data using generative artificial intelligence technology, means for statistically analyzing the extracted data and classifying it by category, means for designing a travel plan based on the classified information and automatically generating a schedule that takes time and budget into consideration, means for generating an optimal travel plan by allowing the user to select images and information that interest them, means for the user to customize the travel plan proposed by the system, and means for transmitting the final travel plan to affiliated travel agencies and providing reservation and payment procedures. This allows the user to easily generate an optimal travel plan and complete the reservation and payment procedures all at once.

[0150] A "social networking service" is a platform that allows users to communicate with each other over the Internet.

[0151] "Means for automatically collecting information" refers to a system that automatically retrieves related posts from social media based on specific keywords or hashtags.

[0152] "Image recognition technology" is a technology that analyzes image data and identifies specific objects or scenes.

[0153] "Tourist destinations" are areas or places that tourists visit, including historical buildings and natural landscapes.

[0154] A "store" is a facility for selling products and providing services.

[0155] "Generative AI technology" is an AI technology that has the ability to learn patterns from large amounts of data and generate new text and content.

[0156] "Text data" refers to information expressed as a string of characters that can be analyzed and searched.

[0157] "Statistical analysis means" refers to a system that uses statistical methods on collected data to analyze its characteristics and trends.

[0158] A "categorization method" is a mechanism for organizing data into categories according to specific criteria, making them easier to identify.

[0159] A "means for designing travel plans" is a system that plans destinations and activities based on the user's interests and collected information.

[0160] A "means for automatically generating a schedule" is a system that uses a program to automatically create a timetable or itinerary based on a designed travel plan.

[0161] "User device" refers to the terminal used by the user to operate or view the site, including smartphones and personal computers.

[0162] "System-proposed travel plans" refers to travel plans automatically generated by the system.

[0163] "Customizable means" means that users can change or adjust the plans suggested by the system to suit their preferences.

[0164] A "travel agent" is a business or organization that provides travel services and sells and books travel packages to customers.

[0165] A "means for providing reservation and payment procedures" is a mechanism that allows users to easily make reservations and complete payments after finalizing their travel plans.

[0166] This invention is a system that automatically collects social media information, analyzes and classifies it using image recognition technology and generative artificial intelligence technology, and provides optimal travel plans based on that information. This system is mainly composed of a server and user terminals. This system is described in detail below.

[0167] System program configuration

[0168] The server automatically collects information from social media sites. Specifically, it uses the social media site's API to automatically retrieve related posts based on specific keywords and hashtags. The collected information includes images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.).

[0169] The collected image data is analyzed using image recognition technology. The server uses a CLIP (Contrastive Language–Image Pretraining) model to identify tourist attractions and stores from the image. For example, if a famous tourist attraction or restaurant appears in the image, it is automatically extracted.

[0170] Next, generative artificial intelligence techniques are used to extract text data: the server uses models such as GPT-3 to extract additional useful information from the captions and comments sections of social media posts, providing the detailed information users crave.

[0171] The extracted data is statistically analyzed and classified into categories such as tourist attractions, restaurants, and events. Based on this, the server designs a travel plan and automatically generates a schedule that takes time and budget into consideration. When a user selects images and information that interest them, the server generates and suggests a travel plan based on that information.

[0172] The travel plan is displayed on the user's device and can be customized by the user. Based on the information adjusted by the user, the server adjusts and automatically generates the travel plan again.

[0173] The finalized itinerary is sent to the partner travel agency, where users can easily purchase and book the itinerary. Specifically, they are redirected to the travel agency's booking page and can complete the payment online.

[0174] Hardware and software used

[0175] Hardware: High-performance servers (e.g., Amazon EC2)

[0176] software:

[0177] API client: requests (for accessing APIs of social media platforms)

[0178] Image Recognition: Transformers (using CLIP model)

[0179] Generative AI: openai (using GPT-3)

[0180] Specific examples

[0181] For example, a user searches for information about "Tokyo travel" and selects a photo that catches their eye. Let's say the photo shows Tokyo Tower. Based on this information, the server generates a schedule that takes into account surrounding tourist spots, restaurants, and transportation options, including Tokyo Tower. If the user wishes to customize the schedule, the server can also accommodate requests such as adding Ueno Zoo and Shibuya.

[0182] Additionally, example prompts have the following format:

[0183] "Generate a travel plan based on the user's interests. Location: Tokyo Tower, Attractions: Ueno Zoo, Shibuya, Restaurant: High-end sushi restaurant, Activities: Sightseeing, Shopping"

[0184] In this way, the server is a system that automatically generates highly accurate travel plans based on the information obtained and provides them to users.

[0185] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0186] Step 1:

[0187] The server automatically collects information from social media sites based on specific keywords and hashtags. This process uses the social media site's API to retrieve images, videos, text, and metadata (location, number of likes, number of comments, etc.). The input is the keywords and hashtags specified by the user, and the output is the collected social media data.

[0188] Step 2:

[0189] The server extracts image data from the collected SNS data and applies image recognition technology to identify tourist attractions and stores. At this stage, the CLIP model is used to analyze the images. The input is image data obtained from SNS, and the output is information on the identified tourist attractions and stores.

[0190] Step 3:

[0191] Based on the identified information, the server uses generative artificial intelligence technology (GPT-3) to extract useful text data from post captions and comments. The input is the image data and text data of tourist attractions and stores identified in Step 2, and the output is the text data generated by the generative AI.

[0192] Step 4:

[0193] The server statistically analyzes the extracted data and classifies it into categories such as tourist attractions, restaurants, and events. The input is text data and metadata, and the output is information classified by category. The analysis uses major statistical methods.

[0194] Step 5:

[0195] The server designs a travel plan based on the classified information and automatically generates a schedule that takes time and budget into consideration. The input is information categorized by category, and the output is a detailed travel schedule. Specifically, it calculates time allocation and expenses by taking into account multiple destinations and activities.

[0196] Step 6:

[0197] The user can view the generated travel plan on their device and select images and information that interest them. The server then generates the optimal travel plan and presents it to the user. The input is the images and information selected by the user, and the output is a customized travel plan.

[0198] Step 7:

[0199] The user can customize the proposed itinerary through the terminal. The server reflects the user's changes and automatically regenerates the adjusted itinerary. The input is the user's customized information, and the output is the adjusted itinerary.

[0200] Step 8:

[0201] The server then sends the finalized itinerary to the affiliated travel agency. The user then completes the procedure to purchase and reserve the itinerary via the terminal. The input is the finalized itinerary, and the output is the data to be sent to the travel agency and a notification of reservation completion.

[0202] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0203] This invention relates to a system that automatically collects information from online social networking services (SNS) and analyzes it using image recognition and generative artificial intelligence technologies to provide users with optimal travel plans. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, it provides users with a more personalized travel experience.

[0204] System configuration

[0205] This system mainly consists of a server, a user's device, and an emotion engine. The server collects information from SNS and analyzes and classifies the data using image recognition and generative artificial intelligence technology. The user's device receives the information, displays the travel plan, and allows for adjustments. The emotion engine also identifies the user's emotions and sends the data to the server.

[0206] Collection and analysis of SNS information

[0207] The server automatically collects posts based on specific keywords and hashtags using the APIs of social media platforms, including images, videos, posting dates and times, poster information, location information, hashtags, number of likes, and number of comments.

[0208] The server applies image recognition technology to the collected images and videos to identify place names, tourist attractions, stores, etc. It also uses generative artificial intelligence technology to extract relevant text data from post captions and comment sections, resulting in information categorized into tourist attractions, restaurants, events, etc.

[0209] Emotion Engine Functions

[0210] The user's device is equipped with a camera and microphone to recognize the user's emotions in real time. The emotion engine analyzes these sensor data and identifies the user's emotions (e.g., joy, surprise, excitement) from facial expressions and tone of voice. The emotion data recognized by the emotion engine is sent to a server for analysis.

[0211] Travel plan design and coordination

[0212] When a user clicks on an image or piece of information that interests them, the device sends that information to the server. Based on this, the server collects information on related tourist spots and stores, and automatically generates a travel plan that takes into account the user's preferences and emotional data. The generated plan also includes transportation and places to stay, and a schedule that takes time and budget into consideration is automatically created.

[0213] For example, if a user clicks on an image of a "trip to Paris" and expresses joy, the server will use that information to suggest a plan that combines popular tourist spots such as the Eiffel Tower and the Louvre.

[0214] View and rearrange your plan

[0215] The device displays the generated itinerary to the user, who can review it and make adjustments as needed, such as changing the order of the stops or selecting additional tourist attractions.

[0216] After the user adjusts the plan, the device sends the information back to the server, which then regenerates the plan based on the user's adjustments and the latest emotion data, and sends the final plan to the device.

[0217] Travel agency collaboration and booking

[0218] The server sends the finalized travel plan to the partner travel agency's system via API, allowing the user to purchase and reserve the travel plan directly. The terminal is then redirected to the travel agency's reservation page, where the user can complete payment procedures, etc. This allows the user to enjoy the optimal travel plan without any hassle.

[0219] As described above, the present invention provides a system that allows users to easily automatically generate optimal travel plans by utilizing information from SNS and combining it with user emotional data.

[0220] The processing flow will be explained below.

[0221] Step 1:

[0222] The server uses the API of a social networking service (SNS) to search for posts based on specific keywords or hashtags (e.g., "travel," "tourist destinations") The data collected through the search includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc.

[0223] Step 2:

[0224] The server stores the collected social media post data in a temporary database, allowing for efficient subsequent analysis. It also stores the metadata of each post for analysis.

[0225] Step 3:

[0226] The server analyzes the collected images using image recognition technology to identify key features (e.g., landmarks, tourist attractions, restaurants), leveraging computer vision techniques to recognize specific landmarks and objects.

[0227] Step 4:

[0228] The server uses generative artificial intelligence techniques to extract relevant text data (e.g., place names, store names, activity details) from post captions and comment sections, and analyzes the extracted text data using natural language processing (NLP) techniques to identify important keywords and phrases.

[0229] Step 5:

[0230] The server statistically analyzes the data extracted through image recognition and text analysis. Specifically, it calculates how frequently each feature appears and evaluates its popularity and relevance. Based on this, the data is classified into categories such as tourist attractions, restaurants, and events.

[0231] Step 6:

[0232] The server creates weekly, monthly, and yearly rankings based on statistically analyzed data. This allows users to easily understand popular spots and activities for each season. This ranking information is also used as reference when creating plans.

[0233] Step 7:

[0234] The camera and microphone on the user's device capture the user's facial expressions and tone of voice, which are then analyzed in real time by the emotion engine.

[0235] Step 8:

[0236] The emotion engine analyzes the user's emotion data (e.g., joy, surprise, excitement) and sends it to the server, where it is linked to the images and information the user is viewing.

[0237] Step 9:

[0238] When a user clicks on an SNS image or information that interests them, the device sends that information and linked emotional data to the server. Based on this, the server collects information on related tourist spots and stores, and automatically generates a travel plan taking into account the user's preference information and emotional data.

[0239] Step 10:

[0240] The server then designs a travel plan based on the user's emotional data, including activities that emphasize fun, based on a strong sense of "joy." The plan also includes transportation and accommodations, and automatically generates a schedule that takes into account time and budget.

[0241] Step 11:

[0242] The device displays the generated itinerary to the user, who can review it and make adjustments as needed, such as changing the order of the stops or selecting additional tourist attractions.

[0243] Step 12:

[0244] After the user adjusts the plan, the device sends the information back to the server, which then regenerates the plan based on the user's adjustments and the latest emotion data, and sends the final plan to the device.

[0245] Step 13:

[0246] The server sends the confirmed travel plan to the partner travel agency's system via API, allowing the user to purchase and reserve the travel plan directly. The terminal is then redirected to the travel agency's reservation page, where the user can complete payment procedures.

[0247] Example 2

[0248] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0249] Conventional travel plan creation systems have difficulty automatically generating personalized plans that reflect the user's emotions and individual preferences. Furthermore, there is a lack of a method for analyzing user emotions in real time and reflecting them in travel plans, making it difficult to improve the quality of the user experience.

[0250] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0251] In this invention, the server includes means for automatically collecting information from social media services on the Internet, means for identifying tourist attractions and stores from the collected information using image recognition technology, means for extracting the identified information as text data using generative artificial intelligence technology, means for collecting and analyzing emotional data in real time using an engine that recognizes user emotions, means for statistically analyzing the extracted data and the collected emotional data and classifying them by category, means for designing a travel plan based on the classified information and analyzed emotional data and automatically generating a schedule that takes time and budget into consideration, and means for displaying the generated travel plan on the user's device so that the user can check and adjust it. This enables the automatic generation of a personalized travel plan that takes into consideration the user's individual preferences and emotions.

[0252] "Internet social media service" refers to an online platform that enables users to share information and interact with others via the Internet.

[0253] "Image recognition technology" refers to the process of automatically identifying and identifying objects and text within an image using computer vision technology.

[0254] "Generative AI technology" refers to AI technology that has the ability to learn patterns from large amounts of data and generate new data and information.

[0255] "User emotion recognition engine" refers to technology that identifies emotions by analyzing a user's facial expressions and tone of voice.

[0256] "Statistical analysis" refers to the process of quantifying large amounts of data and analyzing its patterns and correlations using statistical methods.

[0257] "Categorizing" refers to the process of grouping collected data based on specific attributes or themes.

[0258] "Designing a travel plan" refers to the process of creating a travel schedule and itinerary by combining tourist attractions, activities, means of transportation, etc.

[0259] "Automatically generating a schedule that takes time and budget into consideration" refers to the process of automatically creating an optimal travel schedule using a computer based on the user's desired time period and budget restrictions.

[0260] "Displaying it on the user's device and allowing the user to check and adjust it" means displaying the generated travel plan on a device such as a smartphone or tablet, allowing the user to check and change its contents.

[0261] The present invention is a system that automatically collects information from social media services on the Internet and analyzes that information using image recognition technology and generative artificial intelligence technology to provide users with optimal travel plans. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system provides a more personalized travel experience. Specific embodiments of the system are described below.

[0262] System configuration

[0263] This system consists of a server, a user's terminal, and an emotion engine.

[0264] Collecting SNS information

[0265] The server automatically collects posts based on specific keywords or hashtags using the APIs of social media platforms, such as Twitter API or Instagram Graph API. The collected data includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, and number of comments.

[0266] Image Recognition and Text Analytics

[0267] The server applies image recognition technology (e.g., Google Cloud Vision API) to the collected images and videos to identify place names, tourist attractions, stores, etc. It then uses generative artificial intelligence technology (e.g., OpenAI's GPT-4 (registered trademark)) to analyze posted captions and comments and extract related text data. This allows it to obtain information categorized into tourist attractions, restaurants, events, etc.

[0268] Emotion Engine Functions

[0269] The user's device is equipped with a camera and microphone, which the emotion engine uses to identify emotions in real time from the user's facial expressions and tone of voice. The emotion engine uses, for example, the Emotion API from Microsoft® Azure®. The collected emotion data is sent to a server and used to plan the itinerary.

[0270] Travel plan design

[0271] When a user clicks on an image or piece of information that interests them, the server uses that information to collect information on related tourist spots and stores. Taking into account the user's emotional data and preference information, the server automatically generates a travel plan. The generated plan includes a schedule that takes into account transportation, places to stay, time, and budget. For example, if a user clicks on an image of a "trip to Paris" and expresses joy, the server will suggest a plan that includes popular tourist spots such as the Eiffel Tower, the Louvre, and the Champs-Élysées.

[0272] View and adjust your plan

[0273] The device displays the generated travel plan to the user. The user can review the plan, change the order of the destinations, or select additional tourist attractions. The adjusted plan information is sent back to the server, and the server regenerates the plan based on the user's adjustments and the latest emotion data, and sends the final plan to the device.

[0274] Book a travel plan

[0275] The server sends the finalized itinerary to the partner travel agency's system via API, and the terminal redirects the user to the travel agency's booking page. The user can then proceed with the procedure to purchase and reserve the itinerary, for example, using Expedia API or Booking.com API.

[0276] Examples and prompts

[0277] For example, "If a user clicks on an image of a trip to Paris and expresses joy, the server will suggest a travel itinerary that includes attractions such as the Eiffel Tower, the Louvre, and the Champs-Élysées."

[0278] An example of a prompt for a generative AI model is, "What is the best plan for a trip to Paris? Please list it in categories of tourist attractions, restaurants, and events."

[0279] The above is a detailed description of the mode for carrying out the invention.

[0280] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0281] Step 1: Collect social media information

[0282] Input: Specific keywords or hashtags

[0283] Specific operation: The server uses the API of the social media platform (e.g., Twitter API, Instagram Graph API) to collect posts that match the specified keywords or hashtags.

[0284] Output: Images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc.

[0285] Step 2: Image Recognition

[0286] Input: Collected image and video data

[0287] Specific operation: The server uses image recognition technology (e.g., Google Cloud Vision API) to identify place names, tourist attractions, stores, etc. from this data.

[0288] Output: Information on identified tourist attractions and stores, and corresponding image and video data

[0289] Step 3: Text analysis

[0290] Input: Collected post captions and comments

[0291] How it works: The server uses generative artificial intelligence techniques (e.g., OpenAI's GPT-4) to extract relevant text data from captions and comments and classify them into categories such as tourist attractions, restaurants, and events.

[0292] Output: Extracted text data and categorical information

[0293] Step 4: Collecting emotion data

[0294] Input: User camera and microphone data

[0295] How it works: The device uses an emotion engine (e.g., Microsoft Azure Emotion API) to identify emotions in real time from the user's facial expressions and tone of voice.

[0296] Output: Collected emotion data, identified emotion types

[0297] Step 5: Analyze the sentiment data

[0298] Input: Collected emotion data

[0299] Specific operation: The server analyzes the collected emotional data to identify the user's current emotional state. Based on this analysis result, it is used to generate a travel plan.

[0300] Output: Parsed emotion data

[0301] Step 6: Generate your travel plan

[0302] Input: Identified tourist destination information, extracted text data, analyzed emotion data, user preference information

[0303] Specific operation: Based on this input data, the server collects information on tourist spots and stores, and automatically generates a travel plan taking into account the user's emotional data and preference information.

[0304] Output: Automatically generated travel plan, schedule information, transportation, and accommodation

[0305] Step 7: View your travel plans

[0306] Input: Auto-generated itinerary

[0307] Specific operation: The terminal displays the generated travel plan to the user so that the user can confirm it.

[0308] Output: Travel plan displayed on the user's device

[0309] Step 8: Adjust your travel plans

[0310] Input: User adjustment instructions

[0311] Specific operation: The user checks the displayed plan, changes the order of the stops, or selects additional tourist attractions. The adjustments are sent to the server via the device.

[0312] Output: Adjusted travel plan instructions

[0313] Step 9: Regenerate the adjusted plan

[0314] Input: Adjusted travel plan instructions, latest sentiment data

[0315] Specific operation: The server generates a new travel plan based on these input data and sends the final plan to the terminal.

[0316] Output: Final itinerary

[0317] Step 10: Book your travel plan

[0318] Input: Final itinerary

[0319] Specific operation: The server sends the finalized travel plan to the partner travel agency's system via API, and the terminal redirects the user to the travel agency's booking page, where the user completes the booking procedure.

[0320] Output: Confirmed reservation information, redirect to reservation completion page

[0321] The above is a detailed description of the specific steps of the program processing of this system.

[0322] (Application example 2)

[0323] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0324] Conventional travel plan providing systems have the problem of being unable to flexibly respond to users' emotions and individual needs. They also lack a means to effectively communicate travel plans to users in an easy-to-understand manner. As a result, users end up spending a lot of time and effort selecting travel destinations and planning their schedules.

[0325] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for automatically collecting information from social networking services on the Internet, means for identifying tourist spots and stores from the collected information using image recognition technology, and means for extracting the identified information as text data using generative artificial intelligence technology. This makes it possible to recognize the user's emotions and personalize the travel plan using the emotion data. Furthermore, by adding means for generating a travel plan based on emotions in video format, the travel plan can be communicated to the user visually and effectively.

[0326] An "online social networking service" is a platform that allows users to share information and communicate with each other via the Internet.

[0327] "Means of automatically collecting information" refers to a method of automatically obtaining posted data from social networking services on the Internet using specific keywords or hashtags through a program.

[0328] "Image recognition technology" is a technology in which a computer analyzes image data and identifies specific objects or landmarks.

[0329] "Generative artificial intelligence technology" is an artificial intelligence technology that analyzes collected data and automatically generates text and data.

[0330] "Means of extracting as text data" refers to a method of extracting relevant text information from collected data using image recognition technology or generative artificial intelligence technology and providing it in a meaningful format.

[0331] "Means of statistical analysis and categorization" refers to a method of analyzing extracted data using statistical analysis and dividing it into categories such as tourist attractions, stores, and events based on their relevance.

[0332] "A means for designing travel plans and automatically generating schedules that take time and budget into consideration" is a method for creating travel plans based on classified information and automatically generating the optimal schedule to meet the user's requirements.

[0333] "Means for displaying on the user's device and allowing the user to review and adjust" refers to a method for displaying the generated travel plan on the user's smartphone or computer, allowing the user to review the plan and make changes as necessary.

[0334] "Means for recognizing a user's emotions and using that emotional data to personalize travel plans" refers to a method for identifying a user's emotions by analyzing their facial expressions and tone of voice, and then using that data to provide travel plans that match the user's preferences.

[0335] The "means for generating emotion-based travel plans in video format" is a method for generating videos and providing them to users based on information about tourist spots and events that take into account the user's emotional data.

[0336] The present invention relates to a system that collects information from social networking services (SNS) and provides travel plans in the form of videos that take into account the user's emotions. Specific embodiments of the system will be described below.

[0337] This system mainly consists of a server, a user's device, and an emotion engine. The server includes a social networking data collection module, a data analysis module, a travel plan generation module, and a video generation module. The user's device is equipped with a camera and microphone and includes an emotion engine for emotion recognition.

[0338] Program Overview

[0339] 1. Collecting social media data

[0340] The server uses APIs from social media platforms to automatically collect data based on specific keywords and hashtags, including images, videos, posting dates and times, poster information, location information, hashtags, number of likes, number of comments, etc. Specific software used includes social media APIs and data collection libraries.

[0341] 2. Data Analysis

[0342] The collected data is then analyzed using image recognition technology (e.g., OpenCV or TENSORFLOW®) to identify tourist attractions and stores. Generative AI technology is then used to extract relevant text data from post captions and comment sections. This allows each tourist attraction and store to be classified into a category.

[0343] 3. Emotional Recognition

[0344] The user's device is equipped with a camera and microphone, and by analyzing these sensor data, the emotion engine recognizes the user's emotions in real time. The emotion engine identifies whether the user is expressing emotions such as joy or surprise from facial expressions and tone of voice. This data is sent to a server and used to customize the travel plan. Specific software includes emotion recognition libraries (e.g., FaceReader and Vokaturi).

[0345] 4. Designing a travel plan

[0346] The server designs a travel plan based on the information the user clicks on and their emotional data. The plan includes sightseeing spots, shops, events, etc., and the schedule is automatically generated taking into account time and budget. The generated plan can be adjusted to suit the user's preferences.

[0347] 5. Video Generation and Display

[0348] The finalized itinerary is then generated in video format by a video generation module. The generated video is sent to the user's device, where it presents the itinerary in a visually easy-to-understand format. Specific software includes a video editing library (e.g., FFmpeg, MoviePy, etc.).

[0349] Specific examples

[0350] For example, if a user expresses interest in a trip to Paris, the system collects data related to Paris from social media and analyzes information on tourist attractions and restaurants. If the user clicks on an image of Paris on the screen and expresses joy, the server will suggest a travel plan that combines popular spots such as the Eiffel Tower, the Louvre, and Montmartre. A video based on the plan is generated and provided to the user.

[0351] Prompt Sentence Examples

[0352] "If the user's analyzed emotion is detected as 'surprise,' generate a video of 'recommended travel destinations full of surprises.' For example, suggest a plan that includes unique experiences such as an underwater hotel or a trapeze."

[0353] As described above, the system of the present invention can provide users with optimal travel plans in video format by combining information collected from SNS with user emotion data.

[0354] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0355] Step 1:

[0356] The server automatically collects data from social networking services on the Internet using keywords and hashtags. Keywords that reflect the user's interests (e.g., "trip to Paris") are used for this input. The collected data includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc. The specific software used to collect the data is the SNS API or data collection library. The server obtains this posted data.

[0357] Step 2:

[0358] The server uses image recognition technology (such as OpenCV or TensorFlow) on the collected data to identify tourist attractions and stores. The input is the image data collected in step 1, and the output is text data in which tourist attractions and stores are identified. Image recognition technology is used to identify tourist attraction landmarks, store logos, etc., and the identified information is extracted as text.

[0359] Step 3:

[0360] The server uses generative AI technology to extract relevant text data from post captions and comment sections. The input is the text data of the social media post, and the output is the analyzed text data. Generative AI technology is used to understand the content of the post and extract information about tourist spots, stores, and events. Specifically, a natural language processing library is used.

[0361] Step 4:

[0362] The server statistically analyzes the extracted text data and classifies it into categories such as tourist attractions, stores, and events. The input is the text data extracted in step 3, and the output is information classified by category. The server analyzes the data using a statistical analysis library and classifies it into each category.

[0363] Step 5:

[0364] The server uses the user's emotional data to design a travel plan and automatically generate a schedule that takes time and budget into consideration. The input is the information classified in step 4 and the user's emotional data. The emotion engine uses the user's camera and microphone to obtain emotional data in real time (e.g., joy, surprise, etc.) and generates a travel plan based on those emotions. Specifically, the emotion engine uses FaceReader and Vokaturi.

[0365] Step 6:

[0366] The server generates a travel plan based on emotions in video format and sends it to the user's device. The input is the travel plan generated in step 5, and the output is a video travel plan. Using a video generation module (e.g., FFmpeg or MoviePy), videos of tourist spots and events according to emotions are created and visually presented to the user.

[0367] Step 7:

[0368] The device displays the generated video to the user, who then reviews the plan and adjusts it as necessary. The input is the video generated in step 6, and the output is the travel plan adjusted by the user. The user can review the travel plan through the device and make adjustments, such as changing the order of the destinations. The adjusted information is sent back to the server, and the plan is regenerated as necessary.

[0369] The above is a description of the specific processing steps of the system that realizes the application example.

[0370] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0371] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0372] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0373] [Second embodiment]

[0374] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0375] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0376] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0377] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0378] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0379] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0380] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0381] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0382] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0383] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0384] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0385] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0386] The present invention is a system that automatically collects information from social networking services (SNS) on the Internet, analyzes the information using image recognition technology and generative artificial intelligence technology, and provides users with optimal travel plans. The following describes in detail the embodiments of the present invention.

[0387] System configuration

[0388] This system mainly consists of a server and a user's device. The server collects information from SNS and analyzes and classifies the data using image recognition and generative artificial intelligence technology. The user's device receives the information, displays the travel plan, and allows adjustments.

[0389] Collecting SNS information

[0390] The server automatically collects posts based on specific keywords and hashtags using the APIs of social media platforms. This includes images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.). For example, it collects information based on hashtags such as "Tokyo travel" and "gourmet."

[0391] Image recognition and text data extraction

[0392] The server applies image recognition technology to the collected images and videos to identify place names, tourist attractions, stores, etc. For example, it automatically extracts famous tourist spots (Tokyo Tower and Sensoji Temple) and restaurants. It also uses generative artificial intelligence technology to extract additional text data from post captions and comment sections.

[0393] Data classification and analysis

[0394] The server statistically analyzes the extracted data using image recognition and text analysis, categorizing it into categories, such as tourist attractions, restaurants, and events, and generates rankings for each category. These rankings are displayed based on weekly, monthly, and yearly metrics based on the collected data.

[0395] Travel plan design

[0396] When a user clicks on an image or piece of information that interests them, the server automatically generates a travel plan based on that information. The plan includes places to visit, places to eat, activities, etc. For example, if a user clicks on an image of "Tokyo Tower," a schedule is automatically created that takes into account nearby tourist attractions, restaurants, and transportation options.

[0397] Schedule and budget management

[0398] The server generates a detailed schedule (timetable and budget) based on the travel plan and displays it on the user's device. The user can check this schedule and make changes as needed. For example, a detailed plan could be provided, such as visiting Sensoji Temple in the morning, having lunch at a famous okonomiyaki restaurant in Asakusa, and then visiting Tokyo Tower.

[0399] Travel agency collaboration and booking

[0400] The server sends the finalized travel plan to the partner travel agency, allowing the user to purchase and reserve the plan. The user's device is redirected to the travel agency's reservation page, where payment procedures are completed. In this way, the user can enjoy the optimal travel plan without any hassle.

[0401] Specific examples

[0402] For example, a user searches for "Kyoto travel" on a major online social networking site and clicks on a photo that catches their eye. This photo may show a famous tourist spot in Kyoto (e.g., Kiyomizu-dera Temple). Based on this information, the server automatically generates a travel plan that includes Kiyomizu-dera Temple and provides a schedule that takes into account nearby tourist spots, restaurants, and transportation options. The user can also review this plan and select additional tourist spots (e.g., Kinkaku-ji Temple) if necessary. Finally, once the plan is finalized, the server sends this information to the travel agency, and the reservation process is completed.

[0403] As described above, the present invention provides a system that utilizes information from SNS to enable users to easily automatically generate optimal travel plans.

[0404] The processing flow will be explained below.

[0405] Step 1:

[0406] The server uses the API of a social networking service (SNS) to search for posts based on specific keywords or hashtags (e.g., "travel," "tourist destinations") The data collected through the search includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc.

[0407] Step 2:

[0408] The server stores the collected social media post data in a temporary database, allowing for efficient subsequent analysis. It also stores the metadata of each post for analysis.

[0409] Step 3:

[0410] The server analyzes the collected images using image recognition technology to identify key features (e.g., landmarks, tourist attractions, restaurants), leveraging computer vision techniques to recognize specific landmarks and objects.

[0411] Step 4:

[0412] The server uses generative artificial intelligence techniques to extract relevant text data (e.g., place names, store names, activity details) from post captions and comment sections, and analyzes the extracted text data using natural language processing (NLP) techniques to identify important keywords and phrases.

[0413] Step 5:

[0414] The server statistically analyzes the data extracted through image recognition and text analysis. Specifically, it calculates how frequently each feature appears and evaluates its popularity and relevance. Based on this, the data is classified into categories such as tourist attractions, restaurants, and events.

[0415] Step 6:

[0416] The server creates weekly, monthly, and yearly rankings based on statistically analyzed data. This allows users to easily understand popular spots and activities for each season. This ranking information is also used as reference when creating plans.

[0417] Step 7:

[0418] When a user clicks on an image or piece of information that interests them on the social networking site, the device sends that information to the server, which then collects information on related tourist spots and stores and automatically generates a travel plan based on the user's input.

[0419] Step 8:

[0420] Based on the information the user clicks, the server designs a travel plan that combines tourist spots, restaurants, activities, etc. The generated plan also includes transportation and accommodations, and the schedule is automatically generated taking into account time and budget.

[0421] Step 9:

[0422] The device displays the generated itinerary to the user, who can review it and make adjustments as needed, such as changing the order of the stops or selecting additional tourist attractions.

[0423] Step 10:

[0424] After the user adjusts the plan, the device sends the information back to the server, which then regenerates the plan based on the user's adjustments and sends the final plan to the device.

[0425] Step 11:

[0426] The server sends the confirmed travel plan to the partner travel agency's system via API, allowing the user to purchase and reserve the travel plan directly. The terminal is then redirected to the travel agency's reservation page, where the user can complete payment procedures.

[0427] Example 1

[0428] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0429] Traditional travel planning requires users to manually gather vast amounts of information and create plans, which is time-consuming and labor-intensive. Furthermore, fragmented information on the Internet makes it difficult to efficiently generate comprehensive travel plans. Furthermore, the process of coordinating travel plans and making reservations is cumbersome, placing a significant burden on users.

[0430] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0431] In this invention, the server includes means for automatically collecting information from social networking services on the Internet, means for identifying tourist attractions and stores from the collected information using image recognition technology, means for extracting the identified information as text data using generative artificial intelligence technology, means for statistically analyzing the extracted data and classifying it by category, means for designing a travel plan based on the classified information and automatically generating a schedule that takes time and budget into consideration, means for displaying the generated travel plan on the user's device so that the user can confirm and adjust it, means for automatically generating a new travel plan based on information selected by the user, and means for transmitting the confirmed travel plan to affiliated travel agencies and providing procedures for the user to reserve and purchase the travel plan. This allows users to efficiently generate, adjust, and book an optimal travel plan without any effort.

[0432] A "social networking service" is an online platform that allows users to exchange information and communicate with each other over the Internet.

[0433] "Image recognition technology" is a technology in which a machine analyzes the content of an image and identifies the objects and scenes contained within it.

[0434] "Generative AI technology" is an AI technology that generates natural language sentences, images, etc. based on input data.

[0435] "Text data" refers to data consisting of character information, including sentences and words.

[0436] "Statistical analysis" means analyzing the characteristics and trends of collected data using statistical methods.

[0437] A "travel plan" refers to a plan of the schedule, destinations, activities, etc. for a trip within a specific period of time.

[0438] A "schedule" is a time-based plan that shows the sequence of activities or events that should occur at a particular time.

[0439] "User Device" means a hardware device used by a User, such as a computer, smartphone, or tablet.

[0440] "Travel Agent" means a business or organization that provides travel services, sells travel products, and processes reservations.

[0441] "Reservation procedure" refers to the action of finalizing travel plans and reserving accommodation and transportation based on those plans.

[0442] The present invention is a system that automatically collects information from social networking services (SNS) on the Internet, analyzes the information using image recognition technology and generative artificial intelligence technology, and provides users with optimal travel plans. A detailed description of an embodiment of the present invention will be given below.

[0443] System configuration

[0444] This system mainly consists of a server and a user's device. The server collects information from SNS and analyzes and classifies the data using image recognition and generative artificial intelligence technology. The user's device receives the information, displays the travel plan, and allows adjustments.

[0445] Collecting SNS information

[0446] The server automatically collects posts based on specific keywords and hashtags using the APIs of social media platforms. This includes images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.). For example, it collects information based on hashtags such as "Tokyo travel" and "gourmet."

[0447] Image recognition and text data extraction

[0448] The server applies image recognition technology to the collected images and videos to identify place names, tourist attractions, stores, etc. For example, it automatically extracts famous tourist spots (e.g., Tokyo Tower and Sensoji Temple) and restaurants. It also uses generative artificial intelligence technology to extract additional text data from post captions and comment sections.

[0449] Data classification and analysis

[0450] The server statistically analyzes the extracted data using image recognition and text analysis, categorizing it into categories, such as tourist attractions, restaurants, and events, and generates rankings for each category. These rankings are displayed based on weekly, monthly, and yearly metrics based on the collected data.

[0451] Travel plan design

[0452] When a user clicks on an image or piece of information that interests them, the server automatically generates a travel plan based on that information. The plan includes places to visit, places to eat, activities, etc. For example, if a user clicks on an image of "Tokyo Tower," a schedule is automatically created that takes into account nearby tourist attractions, restaurants, and transportation options.

[0453] Schedule and budget management

[0454] The server generates a detailed schedule (timetable and budget) based on the travel plan and displays it on the user's device. The user can check this schedule and make changes as needed. For example, a detailed plan could be provided, such as visiting Sensoji Temple in the morning, having lunch at a famous okonomiyaki restaurant in Asakusa, and then visiting Tokyo Tower.

[0455] Travel agency collaboration and booking

[0456] The server sends the finalized travel plan to the partner travel agency, allowing the user to purchase and reserve the plan. The user's device is redirected to the travel agency's reservation page, where payment procedures are completed. In this way, the user can enjoy the optimal travel plan without any hassle.

[0457] Specific examples

[0458] For example, a user searches for "Kyoto travel" on a major online social networking site and clicks on a photo that catches their eye. This photo may show a famous tourist spot in Kyoto (e.g., Kiyomizu-dera Temple). Based on this information, the server automatically generates a travel plan that includes Kiyomizu-dera Temple and provides a schedule that takes into account nearby tourist spots, restaurants, and transportation options. The user can also review this plan and select additional tourist spots (e.g., Kinkaku-ji Temple) if necessary. Finally, once the plan is finalized, the server sends this information to the travel agency, and the reservation process is completed.

[0459] Prompt Sentence Examples

[0460] "A user searched for 'Kyoto travel' on a major online social networking site and clicked on a photo that caught their eye (Kiyomizu-dera Temple). Based on this information, please create a Kyoto trip plan that includes Kiyomizu-dera Temple. Please provide a detailed itinerary that takes into account places to visit, restaurants, recommended activities, and transportation options."

[0461] In this way, the present invention provides a system that utilizes information from SNS to enable users to easily automatically generate optimal travel plans.

[0462] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0463] Step 1:

[0464] The server automatically collects social media information. It uses the API of social media platforms, such as the Twitter API, to collect posts based on specific keywords and hashtags (for example, "Tokyo travel" or "food"). The input is the keywords and hashtags specified by the user. The output is the collected images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.). Specifically, the server calls the API to retrieve posts in real time and stores them in a database.

[0465] Step 2:

[0466] The server performs image recognition on the collected images and videos. It uses the Google Cloud Vision API to identify tourist attractions and stores contained in the images. The input is the collected images and videos. The output is information about the identified tourist attractions and stores (e.g., Tokyo Tower, Sensoji Temple). Specifically, the server calls the image recognition API, analyzes objects and text in the images, and stores the results in a database.

[0467] Step 3:

[0468] The server extracts text data using generative artificial intelligence technology. It uses a generative AI model, such as OpenAI's GPT-3.5, to generate text data from the collected post captions and comments. The input is the post captions and comments. The output is the extracted text data. Specifically, the server sends prompts to the generative AI model, analyzes the generated text, and extracts important information.

[0469] Step 4:

[0470] The server statistically analyzes the extracted data and classifies it into categories, such as tourist attractions, restaurants, and events, and generates rankings for each category. The input is information about identified tourist attractions and stores and the extracted text data. The output is the data classified by category and rankings. Specifically, the server retrieves data from the database, applies statistical analysis algorithms to classify the data, and calculates rankings.

[0471] Step 5:

[0472] When a user clicks on an image or piece of information that interests them, the server automatically generates a travel plan based on that information. The input is the images and information selected by the user. The output is an automatically generated travel plan (places to visit, places to eat, activities, etc.). Specifically, the server runs a travel plan generation algorithm based on the selected data to create an optimal schedule for each tourist spot and shop.

[0473] Step 6:

[0474] The server generates a detailed schedule (timetable and budget) based on the travel plan and displays it on the user's device. The input is the automatically generated travel plan. The output is the detailed schedule and budget. Specifically, the server analyzes the travel plan and applies a time and budget management algorithm to determine a specific schedule and sends it to the user's device.

[0475] Step 7:

[0476] The server sends the final confirmed travel plan to the partner travel agency, allowing the user to purchase and book the travel plan. The input is the confirmed travel plan. The output is a notification of completion of transmission to the travel agency and the progress of the reservation. Specifically, the server calls the partner agency's API to send the travel plan and manages the reservation and purchase procedures. On the user's device, the server is redirected to the reservation page and the payment procedure is carried out.

[0477] Through the above steps, this system utilizes information from SNS to enable users to easily automatically generate and book optimal travel plans.

[0478] (Application example 1)

[0479] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0480] Conventional travel planning systems require users to manually search for information on tourist spots, restaurants, and other attractions that interest them and then create a plan based on that information, which is both time-consuming and laborious. Furthermore, generating an optimal travel plan requires integrating a large amount of information, which places a significant burden on users. Furthermore, reservations and payment procedures must be completed separately, which is time-consuming. To solve these issues, a system is needed that automatically collects information that interests users and provides optimal travel plans.

[0481] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0482] In this invention, the server includes means for automatically collecting information from social networking services on the Internet, means for identifying tourist attractions and stores from the collected information using image recognition technology, means for extracting the identified information as text data using generative artificial intelligence technology, means for statistically analyzing the extracted data and classifying it by category, means for designing a travel plan based on the classified information and automatically generating a schedule that takes time and budget into consideration, means for generating an optimal travel plan by allowing the user to select images and information that interest them, means for the user to customize the travel plan proposed by the system, and means for transmitting the final travel plan to affiliated travel agencies and providing reservation and payment procedures. This allows the user to easily generate an optimal travel plan and complete the reservation and payment procedures all at once.

[0483] A "social networking service" is a platform that allows users to communicate with each other over the Internet.

[0484] "Means for automatically collecting information" refers to a system that automatically retrieves related posts from social media based on specific keywords or hashtags.

[0485] "Image recognition technology" is a technology that analyzes image data and identifies specific objects or scenes.

[0486] "Tourist destinations" are areas or places that tourists visit, including historical buildings and natural landscapes.

[0487] A "store" is a facility for selling products and providing services.

[0488] "Generative AI technology" is an AI technology that has the ability to learn patterns from large amounts of data and generate new text and content.

[0489] "Text data" refers to information expressed as a string of characters that can be analyzed and searched.

[0490] "Statistical analysis means" refers to a system that uses statistical methods on collected data to analyze its characteristics and trends.

[0491] A "categorization method" is a mechanism for organizing data into categories according to specific criteria, making them easier to identify.

[0492] A "means for designing travel plans" is a system that plans destinations and activities based on the user's interests and collected information.

[0493] A "means for automatically generating a schedule" is a system that uses a program to automatically create a timetable or itinerary based on a designed travel plan.

[0494] "User device" refers to the terminal used by the user to operate or view the site, including smartphones and personal computers.

[0495] "System-proposed travel plans" refers to travel plans automatically generated by the system.

[0496] "Customizable means" means that users can change or adjust the plans suggested by the system to suit their preferences.

[0497] A "travel agent" is a business or organization that provides travel services and sells and books travel packages to customers.

[0498] A "means for providing reservation and payment procedures" is a mechanism that allows users to easily make reservations and complete payments after finalizing their travel plans.

[0499] This invention is a system that automatically collects social media information, analyzes and classifies it using image recognition technology and generative artificial intelligence technology, and provides optimal travel plans based on that information. This system is mainly composed of a server and user terminals. This system is described in detail below.

[0500] System program configuration

[0501] The server automatically collects information from social media sites. Specifically, it uses the social media site's API to automatically retrieve related posts based on specific keywords and hashtags. The collected information includes images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.).

[0502] The collected image data is analyzed using image recognition technology. The server uses a CLIP (Contrastive Language–Image Pretraining) model to identify tourist attractions and stores from the image. For example, if a famous tourist attraction or restaurant appears in the image, it is automatically extracted.

[0503] Next, generative artificial intelligence techniques are used to extract text data: the server uses models such as GPT-3 to extract additional useful information from the captions and comments sections of social media posts, providing the detailed information users crave.

[0504] The extracted data is statistically analyzed and classified into categories such as tourist attractions, restaurants, and events. Based on this, the server designs a travel plan and automatically generates a schedule that takes time and budget into consideration. When a user selects images and information that interest them, the server generates and suggests a travel plan based on that information.

[0505] The travel plan is displayed on the user's device and can be customized by the user. Based on the information adjusted by the user, the server adjusts and automatically generates the travel plan again.

[0506] The finalized itinerary is sent to the partner travel agency, where users can easily purchase and book the itinerary. Specifically, they are redirected to the travel agency's booking page and can complete the payment online.

[0507] Hardware and software used

[0508] Hardware: High-performance servers (e.g., Amazon EC2)

[0509] software:

[0510] API client: requests (for accessing APIs of social media platforms)

[0511] Image Recognition: Transformers (using CLIP model)

[0512] Generative AI: openai (using GPT-3)

[0513] Specific examples

[0514] For example, a user searches for information about "Tokyo travel" and selects a photo that catches their eye. Let's say the photo shows Tokyo Tower. Based on this information, the server generates a schedule that takes into account surrounding tourist spots, restaurants, and transportation options, including Tokyo Tower. If the user wishes to customize the schedule, the server can also accommodate requests such as adding Ueno Zoo and Shibuya.

[0515] Additionally, example prompts have the following format:

[0516] "Generate a travel plan based on the user's interests. Location: Tokyo Tower, Attractions: Ueno Zoo, Shibuya, Restaurant: High-end sushi restaurant, Activities: Sightseeing, Shopping"

[0517] In this way, the server is a system that automatically generates highly accurate travel plans based on the information obtained and provides them to users.

[0518] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0519] Step 1:

[0520] The server automatically collects information from social media sites based on specific keywords and hashtags. This process uses the social media site's API to retrieve images, videos, text, and metadata (location, number of likes, number of comments, etc.). The input is the keywords and hashtags specified by the user, and the output is the collected social media data.

[0521] Step 2:

[0522] The server extracts image data from the collected SNS data and applies image recognition technology to identify tourist attractions and stores. At this stage, the CLIP model is used to analyze the images. The input is image data obtained from SNS, and the output is information on the identified tourist attractions and stores.

[0523] Step 3:

[0524] Based on the identified information, the server uses generative artificial intelligence technology (GPT-3) to extract useful text data from post captions and comments. The input is the image data and text data of tourist attractions and stores identified in Step 2, and the output is the text data generated by the generative AI.

[0525] Step 4:

[0526] The server statistically analyzes the extracted data and classifies it into categories such as tourist attractions, restaurants, and events. The input is text data and metadata, and the output is information classified by category. The analysis uses major statistical methods.

[0527] Step 5:

[0528] The server designs a travel plan based on the classified information and automatically generates a schedule that takes time and budget into consideration. The input is information categorized by category, and the output is a detailed travel schedule. Specifically, it calculates time allocation and expenses by taking into account multiple destinations and activities.

[0529] Step 6:

[0530] The user can view the generated travel plan on their device and select images and information that interest them. The server then generates the optimal travel plan and presents it to the user. The input is the images and information selected by the user, and the output is a customized travel plan.

[0531] Step 7:

[0532] The user can customize the proposed itinerary through the terminal. The server reflects the user's changes and automatically regenerates the adjusted itinerary. The input is the user's customized information, and the output is the adjusted itinerary.

[0533] Step 8:

[0534] The server then sends the finalized itinerary to the affiliated travel agency. The user then completes the procedure to purchase and reserve the itinerary via the terminal. The input is the finalized itinerary, and the output is the data to be sent to the travel agency and a notification of reservation completion.

[0535] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0536] This invention relates to a system that automatically collects information from online social networking services (SNS) and analyzes it using image recognition and generative artificial intelligence technologies to provide users with optimal travel plans. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, it provides users with a more personalized travel experience.

[0537] System configuration

[0538] This system mainly consists of a server, a user's device, and an emotion engine. The server collects information from SNS and analyzes and classifies the data using image recognition and generative artificial intelligence technology. The user's device receives the information, displays the travel plan, and allows for adjustments. The emotion engine also identifies the user's emotions and sends the data to the server.

[0539] Collection and analysis of SNS information

[0540] The server automatically collects posts based on specific keywords and hashtags using the APIs of social media platforms, including images, videos, posting dates and times, poster information, location information, hashtags, number of likes, and number of comments.

[0541] The server applies image recognition technology to the collected images and videos to identify place names, tourist attractions, stores, etc. It also uses generative artificial intelligence technology to extract relevant text data from post captions and comment sections, resulting in information categorized into tourist attractions, restaurants, events, etc.

[0542] Emotion Engine Functions

[0543] The user's device is equipped with a camera and microphone to recognize the user's emotions in real time. The emotion engine analyzes these sensor data and identifies the user's emotions (e.g., joy, surprise, excitement) from facial expressions and tone of voice. The emotion data recognized by the emotion engine is sent to a server for analysis.

[0544] Travel plan design and coordination

[0545] When a user clicks on an image or piece of information that interests them, the device sends that information to the server. Based on this, the server collects information on related tourist spots and stores, and automatically generates a travel plan that takes into account the user's preferences and emotional data. The generated plan also includes transportation and places to stay, and a schedule that takes time and budget into consideration is automatically created.

[0546] For example, if a user clicks on an image of a "trip to Paris" and expresses joy, the server will use that information to suggest a plan that combines popular tourist spots such as the Eiffel Tower and the Louvre.

[0547] View and rearrange your plan

[0548] The device displays the generated itinerary to the user, who can review it and make adjustments as needed, such as changing the order of the stops or selecting additional tourist attractions.

[0549] After the user adjusts the plan, the device sends the information back to the server, which then regenerates the plan based on the user's adjustments and the latest emotion data, and sends the final plan to the device.

[0550] Travel agency collaboration and booking

[0551] The server sends the finalized travel plan to the partner travel agency's system via API, allowing the user to purchase and reserve the travel plan directly. The terminal is then redirected to the travel agency's reservation page, where the user can complete payment procedures, etc. This allows the user to enjoy the optimal travel plan without any hassle.

[0552] As described above, the present invention provides a system that allows users to easily automatically generate optimal travel plans by utilizing information from SNS and combining it with user emotional data.

[0553] The processing flow will be explained below.

[0554] Step 1:

[0555] The server uses the API of a social networking service (SNS) to search for posts based on specific keywords or hashtags (e.g., "travel," "tourist destinations") The data collected through the search includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc.

[0556] Step 2:

[0557] The server stores the collected social media post data in a temporary database, allowing for efficient subsequent analysis. It also stores the metadata of each post for analysis.

[0558] Step 3:

[0559] The server analyzes the collected images using image recognition technology to identify key features (e.g., landmarks, tourist attractions, restaurants), leveraging computer vision techniques to recognize specific landmarks and objects.

[0560] Step 4:

[0561] The server uses generative artificial intelligence techniques to extract relevant text data (e.g., place names, store names, activity details) from post captions and comment sections, and analyzes the extracted text data using natural language processing (NLP) techniques to identify important keywords and phrases.

[0562] Step 5:

[0563] The server statistically analyzes the data extracted through image recognition and text analysis. Specifically, it calculates how frequently each feature appears and evaluates its popularity and relevance. Based on this, the data is classified into categories such as tourist attractions, restaurants, and events.

[0564] Step 6:

[0565] The server creates weekly, monthly, and yearly rankings based on statistically analyzed data. This allows users to easily understand popular spots and activities for each season. This ranking information is also used as reference when creating plans.

[0566] Step 7:

[0567] The camera and microphone on the user's device capture the user's facial expressions and tone of voice, which are then analyzed in real time by the emotion engine.

[0568] Step 8:

[0569] The emotion engine analyzes the user's emotion data (e.g., joy, surprise, excitement) and sends it to the server, where it is linked to the images and information the user is viewing.

[0570] Step 9:

[0571] When a user clicks on an SNS image or information that interests them, the device sends that information and linked emotional data to the server. Based on this, the server collects information on related tourist spots and stores, and automatically generates a travel plan taking into account the user's preference information and emotional data.

[0572] Step 10:

[0573] The server then designs a travel plan based on the user's emotional data, including activities that emphasize fun, based on a strong sense of "joy." The plan also includes transportation and accommodations, and automatically generates a schedule that takes into account time and budget.

[0574] Step 11:

[0575] The device displays the generated itinerary to the user, who can review it and make adjustments as needed, such as changing the order of the stops or selecting additional tourist attractions.

[0576] Step 12:

[0577] After the user adjusts the plan, the device sends the information back to the server, which then regenerates the plan based on the user's adjustments and the latest emotion data, and sends the final plan to the device.

[0578] Step 13:

[0579] The server sends the confirmed travel plan to the partner travel agency's system via API, allowing the user to purchase and reserve the travel plan directly. The terminal is then redirected to the travel agency's reservation page, where the user can complete payment procedures.

[0580] Example 2

[0581] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0582] Conventional travel plan creation systems have difficulty automatically generating personalized plans that reflect the user's emotions and individual preferences. Furthermore, there is a lack of a method for analyzing user emotions in real time and reflecting them in travel plans, making it difficult to improve the quality of the user experience.

[0583] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0584] In this invention, the server includes means for automatically collecting information from social media services on the Internet, means for identifying tourist attractions and stores from the collected information using image recognition technology, means for extracting the identified information as text data using generative artificial intelligence technology, means for collecting and analyzing emotional data in real time using an engine that recognizes user emotions, means for statistically analyzing the extracted data and the collected emotional data and classifying them by category, means for designing a travel plan based on the classified information and analyzed emotional data and automatically generating a schedule that takes time and budget into consideration, and means for displaying the generated travel plan on the user's device so that the user can check and adjust it. This enables the automatic generation of a personalized travel plan that takes into consideration the user's individual preferences and emotions.

[0585] "Internet social media service" refers to an online platform that enables users to share information and interact with others via the Internet.

[0586] "Image recognition technology" refers to the process of automatically identifying and identifying objects and text within an image using computer vision technology.

[0587] "Generative AI technology" refers to AI technology that has the ability to learn patterns from large amounts of data and generate new data and information.

[0588] "User emotion recognition engine" refers to technology that identifies emotions by analyzing a user's facial expressions and tone of voice.

[0589] "Statistical analysis" refers to the process of quantifying large amounts of data and analyzing its patterns and correlations using statistical methods.

[0590] "Categorizing" refers to the process of grouping collected data based on specific attributes or themes.

[0591] "Designing a travel plan" refers to the process of creating a travel schedule and itinerary by combining tourist attractions, activities, means of transportation, etc.

[0592] "Automatically generating a schedule that takes time and budget into consideration" refers to the process of automatically creating an optimal travel schedule using a computer based on the user's desired time period and budget restrictions.

[0593] "Displaying it on the user's device and allowing the user to check and adjust it" means displaying the generated travel plan on a device such as a smartphone or tablet, allowing the user to check and change its contents.

[0594] The present invention is a system that automatically collects information from social media services on the Internet and analyzes that information using image recognition technology and generative artificial intelligence technology to provide users with optimal travel plans. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system provides a more personalized travel experience. Specific embodiments of the system are described below.

[0595] System configuration

[0596] This system consists of a server, a user's terminal, and an emotion engine.

[0597] Collecting SNS information

[0598] The server automatically collects posts based on specific keywords or hashtags using the APIs of social media platforms, such as Twitter API or Instagram Graph API. The collected data includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, and number of comments.

[0599] Image Recognition and Text Analytics

[0600] The server applies image recognition technology (e.g., Google Cloud Vision API) to the collected images and videos to identify place names, tourist attractions, stores, etc. It then uses generative artificial intelligence technology (e.g., OpenAI's GPT-4) to analyze posted captions and comments and extract related text data. This allows it to obtain information categorized into tourist attractions, restaurants, events, etc.

[0601] Emotion Engine Functions

[0602] The user's device is equipped with a camera and microphone, which the emotion engine uses to identify emotions in real time from the user's facial expressions and tone of voice. The emotion engine uses, for example, Microsoft Azure's Emotion API. The collected emotion data is sent to a server and used to plan the itinerary.

[0603] Travel plan design

[0604] When a user clicks on an image or piece of information that interests them, the server uses that information to collect information on related tourist spots and stores. Taking into account the user's emotional data and preference information, the server automatically generates a travel plan. The generated plan includes a schedule that takes into account transportation, places to stay, time, and budget. For example, if a user clicks on an image of a "trip to Paris" and expresses joy, the server will suggest a plan that includes popular tourist spots such as the Eiffel Tower, the Louvre, and the Champs-Élysées.

[0605] View and adjust your plan

[0606] The device displays the generated travel plan to the user. The user can review the plan, change the order of the destinations, or select additional tourist attractions. The adjusted plan information is sent back to the server, and the server regenerates the plan based on the user's adjustments and the latest emotion data, and sends the final plan to the device.

[0607] Book a travel plan

[0608] The server sends the finalized itinerary to the partner travel agency's system via API, and the terminal redirects the user to the travel agency's booking page. The user can then proceed with the procedure to purchase and reserve the itinerary, for example, using Expedia API or Booking.com API.

[0609] Examples and prompts

[0610] For example, "If a user clicks on an image of a trip to Paris and expresses joy, the server will suggest a travel itinerary that includes attractions such as the Eiffel Tower, the Louvre, and the Champs-Élysées."

[0611] An example of a prompt for a generative AI model is, "What is the best plan for a trip to Paris? Please list it in categories of tourist attractions, restaurants, and events."

[0612] The above is a detailed description of the mode for carrying out the invention.

[0613] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0614] Step 1: Collect social media information

[0615] Input: Specific keywords or hashtags

[0616] Specific operation: The server uses the API of the social media platform (e.g., Twitter API, Instagram Graph API) to collect posts that match the specified keywords or hashtags.

[0617] Output: Images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc.

[0618] Step 2: Image Recognition

[0619] Input: Collected image and video data

[0620] Specific operation: The server uses image recognition technology (e.g., Google Cloud Vision API) to identify place names, tourist attractions, stores, etc. from this data.

[0621] Output: Information on identified tourist attractions and stores, and corresponding image and video data

[0622] Step 3: Text analysis

[0623] Input: Collected post captions and comments

[0624] How it works: The server uses generative artificial intelligence techniques (e.g., OpenAI's GPT-4) to extract relevant text data from captions and comments and classify them into categories such as tourist attractions, restaurants, and events.

[0625] Output: Extracted text data and categorical information

[0626] Step 4: Collecting emotion data

[0627] Input: User camera and microphone data

[0628] How it works: The device uses an emotion engine (e.g., Microsoft Azure Emotion API) to identify emotions in real time from the user's facial expressions and tone of voice.

[0629] Output: Collected emotion data, identified emotion types

[0630] Step 5: Analyze the sentiment data

[0631] Input: Collected emotion data

[0632] Specific operation: The server analyzes the collected emotional data to identify the user's current emotional state. Based on this analysis result, it is used to generate a travel plan.

[0633] Output: Parsed emotion data

[0634] Step 6: Generate your travel plan

[0635] Input: Identified tourist destination information, extracted text data, analyzed emotion data, user preference information

[0636] Specific operation: Based on this input data, the server collects information on tourist spots and stores, and automatically generates a travel plan taking into account the user's emotional data and preference information.

[0637] Output: Automatically generated travel plan, schedule information, transportation, and accommodation

[0638] Step 7: View your travel plans

[0639] Input: Auto-generated itinerary

[0640] Specific operation: The terminal displays the generated travel plan to the user so that the user can confirm it.

[0641] Output: Travel plan displayed on the user's device

[0642] Step 8: Adjust your travel plans

[0643] Input: User adjustment instructions

[0644] Specific operation: The user checks the displayed plan, changes the order of the stops, or selects additional tourist attractions. The adjustments are sent to the server via the device.

[0645] Output: Adjusted travel plan instructions

[0646] Step 9: Regenerate the adjusted plan

[0647] Input: Adjusted travel plan instructions, latest sentiment data

[0648] Specific operation: The server generates a new travel plan based on these input data and sends the final plan to the terminal.

[0649] Output: Final itinerary

[0650] Step 10: Book your travel plan

[0651] Input: Final itinerary

[0652] Specific operation: The server sends the finalized travel plan to the partner travel agency's system via API, and the terminal redirects the user to the travel agency's booking page, where the user completes the booking procedure.

[0653] Output: Confirmed reservation information, redirect to reservation completion page

[0654] The above is a detailed description of the specific steps of the program processing of this system.

[0655] (Application example 2)

[0656] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0657] Conventional travel plan providing systems have the problem of being unable to flexibly respond to users' emotions and individual needs. They also lack a means to effectively communicate travel plans to users in an easy-to-understand manner. As a result, users end up spending a lot of time and effort selecting travel destinations and planning their schedules.

[0658] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for automatically collecting information from social networking services on the Internet, means for identifying tourist spots and stores from the collected information using image recognition technology, and means for extracting the identified information as text data using generative artificial intelligence technology. This makes it possible to recognize the user's emotions and personalize the travel plan using the emotion data. Furthermore, by adding means for generating a travel plan based on emotions in video format, the travel plan can be communicated to the user visually and effectively.

[0659] An "online social networking service" is a platform that allows users to share information and communicate with each other via the Internet.

[0660] "Means of automatically collecting information" refers to a method of automatically obtaining posted data from social networking services on the Internet using specific keywords or hashtags through a program.

[0661] "Image recognition technology" is a technology in which a computer analyzes image data and identifies specific objects or landmarks.

[0662] "Generative artificial intelligence technology" is an artificial intelligence technology that analyzes collected data and automatically generates text and data.

[0663] "Means of extracting as text data" refers to a method of extracting relevant text information from collected data using image recognition technology or generative artificial intelligence technology and providing it in a meaningful format.

[0664] "Means of statistical analysis and categorization" refers to a method of analyzing extracted data using statistical analysis and dividing it into categories such as tourist attractions, stores, and events based on their relevance.

[0665] "A means for designing travel plans and automatically generating schedules that take time and budget into consideration" is a method for creating travel plans based on classified information and automatically generating the optimal schedule to meet the user's requirements.

[0666] "Means for displaying on the user's device and allowing the user to review and adjust" refers to a method for displaying the generated travel plan on the user's smartphone or computer, allowing the user to review the plan and make changes as necessary.

[0667] "Means for recognizing a user's emotions and using that emotional data to personalize travel plans" refers to a method for identifying a user's emotions by analyzing their facial expressions and tone of voice, and then using that data to provide travel plans that match the user's preferences.

[0668] The "means for generating emotion-based travel plans in video format" is a method for generating videos and providing them to users based on information about tourist spots and events that take into account the user's emotional data.

[0669] The present invention relates to a system that collects information from social networking services (SNS) and provides travel plans in the form of videos that take into account the user's emotions. Specific embodiments of the system will be described below.

[0670] This system mainly consists of a server, a user's device, and an emotion engine. The server includes a social networking data collection module, a data analysis module, a travel plan generation module, and a video generation module. The user's device is equipped with a camera and microphone and includes an emotion engine for emotion recognition.

[0671] Program Overview

[0672] 1. Collecting social media data

[0673] The server uses APIs from social media platforms to automatically collect data based on specific keywords and hashtags, including images, videos, posting dates and times, poster information, location information, hashtags, number of likes, number of comments, etc. Specific software used includes social media APIs and data collection libraries.

[0674] 2. Data Analysis

[0675] The collected data is then analyzed using image recognition technology (e.g., OpenCV and TensorFlow) to identify tourist attractions and stores. Generative AI technology is then used to extract relevant text data from post captions and comment sections, allowing each tourist attraction and store to be classified into a category.

[0676] 3. Emotional Recognition

[0677] The user's device is equipped with a camera and microphone, and by analyzing these sensor data, the emotion engine recognizes the user's emotions in real time. The emotion engine identifies whether the user is expressing emotions such as joy or surprise from facial expressions and tone of voice. This data is sent to a server and used to customize the travel plan. Specific software includes emotion recognition libraries (e.g., FaceReader and Vokaturi).

[0678] 4. Designing a travel plan

[0679] The server designs a travel plan based on the information the user clicks on and their emotional data. The plan includes sightseeing spots, shops, events, etc., and the schedule is automatically generated taking into account time and budget. The generated plan can be adjusted to suit the user's preferences.

[0680] 5. Video Generation and Display

[0681] The finalized itinerary is then generated in video format by a video generation module. The generated video is sent to the user's device, where it presents the itinerary in a visually easy-to-understand format. Specific software includes a video editing library (e.g., FFmpeg, MoviePy, etc.).

[0682] Specific examples

[0683] For example, if a user expresses interest in a trip to Paris, the system collects data related to Paris from social media and analyzes information on tourist attractions and restaurants. If the user clicks on an image of Paris on the screen and expresses joy, the server will suggest a travel plan that combines popular spots such as the Eiffel Tower, the Louvre, and Montmartre. A video based on the plan is generated and provided to the user.

[0684] Prompt Sentence Examples

[0685] "If the user's analyzed emotion is detected as 'surprise,' generate a video of 'recommended travel destinations full of surprises.' For example, suggest a plan that includes unique experiences such as an underwater hotel or a trapeze."

[0686] As described above, the system of the present invention can provide users with optimal travel plans in video format by combining information collected from SNS with user emotion data.

[0687] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0688] Step 1:

[0689] The server automatically collects data from social networking services on the Internet using keywords and hashtags. Keywords that reflect the user's interests (e.g., "trip to Paris") are used for this input. The collected data includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc. The specific software used to collect the data is the SNS API or data collection library. The server obtains this posted data.

[0690] Step 2:

[0691] The server uses image recognition technology (such as OpenCV or TensorFlow) on the collected data to identify tourist attractions and stores. The input is the image data collected in step 1, and the output is text data in which tourist attractions and stores are identified. Image recognition technology is used to identify tourist attraction landmarks, store logos, etc., and the identified information is extracted as text.

[0692] Step 3:

[0693] The server uses generative AI technology to extract relevant text data from post captions and comment sections. The input is the text data of the social media post, and the output is the analyzed text data. Generative AI technology is used to understand the content of the post and extract information about tourist spots, stores, and events. Specifically, a natural language processing library is used.

[0694] Step 4:

[0695] The server statistically analyzes the extracted text data and classifies it into categories such as tourist attractions, stores, and events. The input is the text data extracted in step 3, and the output is information classified by category. The server analyzes the data using a statistical analysis library and classifies it into each category.

[0696] Step 5:

[0697] The server uses the user's emotional data to design a travel plan and automatically generate a schedule that takes time and budget into consideration. The input is the information classified in step 4 and the user's emotional data. The emotion engine uses the user's camera and microphone to obtain emotional data in real time (e.g., joy, surprise, etc.) and generates a travel plan based on those emotions. Specifically, the emotion engine uses FaceReader and Vokaturi.

[0698] Step 6:

[0699] The server generates a travel plan based on emotions in video format and sends it to the user's device. The input is the travel plan generated in step 5, and the output is a video travel plan. Using a video generation module (e.g., FFmpeg or MoviePy), videos of tourist spots and events according to emotions are created and visually presented to the user.

[0700] Step 7:

[0701] The device displays the generated video to the user, who then reviews the plan and adjusts it as necessary. The input is the video generated in step 6, and the output is the travel plan adjusted by the user. The user can review the travel plan through the device and make adjustments, such as changing the order of the destinations. The adjusted information is sent back to the server, and the plan is regenerated as necessary.

[0702] The above is a description of the specific processing steps of the system that realizes the application example.

[0703] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0704] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0705] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0706] [Third embodiment]

[0707] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0708] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0709] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0710] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0711] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0712] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0713] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0714] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0715] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0716] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0717] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0718] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0719] The present invention is a system that automatically collects information from social networking services (SNS) on the Internet, analyzes the information using image recognition technology and generative artificial intelligence technology, and provides users with optimal travel plans. The following describes in detail the embodiments of the present invention.

[0720] System configuration

[0721] This system mainly consists of a server and a user's device. The server collects information from SNS and analyzes and classifies the data using image recognition and generative artificial intelligence technology. The user's device receives the information, displays the travel plan, and allows adjustments.

[0722] Collecting SNS information

[0723] The server automatically collects posts based on specific keywords and hashtags using the APIs of social media platforms. This includes images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.). For example, it collects information based on hashtags such as "Tokyo travel" and "gourmet."

[0724] Image recognition and text data extraction

[0725] The server applies image recognition technology to the collected images and videos to identify place names, tourist attractions, stores, etc. For example, it automatically extracts famous tourist spots (Tokyo Tower and Sensoji Temple) and restaurants. It also uses generative artificial intelligence technology to extract additional text data from post captions and comment sections.

[0726] Data classification and analysis

[0727] The server statistically analyzes the extracted data using image recognition and text analysis, categorizing it into categories, such as tourist attractions, restaurants, and events, and generates rankings for each category. These rankings are displayed based on weekly, monthly, and yearly metrics based on the collected data.

[0728] Travel plan design

[0729] When a user clicks on an image or piece of information that interests them, the server automatically generates a travel plan based on that information. The plan includes places to visit, places to eat, activities, etc. For example, if a user clicks on an image of "Tokyo Tower," a schedule is automatically created that takes into account nearby tourist attractions, restaurants, and transportation options.

[0730] Schedule and budget management

[0731] The server generates a detailed schedule (timetable and budget) based on the travel plan and displays it on the user's device. The user can check this schedule and make changes as needed. For example, a detailed plan could be provided, such as visiting Sensoji Temple in the morning, having lunch at a famous okonomiyaki restaurant in Asakusa, and then visiting Tokyo Tower.

[0732] Travel agency collaboration and booking

[0733] The server sends the finalized travel plan to the partner travel agency, allowing the user to purchase and reserve the plan. The user's device is redirected to the travel agency's reservation page, where payment procedures are completed. In this way, the user can enjoy the optimal travel plan without any hassle.

[0734] Specific examples

[0735] For example, a user searches for "Kyoto travel" on a major online social networking site and clicks on a photo that catches their eye. This photo may show a famous tourist spot in Kyoto (e.g., Kiyomizu-dera Temple). Based on this information, the server automatically generates a travel plan that includes Kiyomizu-dera Temple and provides a schedule that takes into account nearby tourist spots, restaurants, and transportation options. The user can also review this plan and select additional tourist spots (e.g., Kinkaku-ji Temple) if necessary. Finally, once the plan is finalized, the server sends this information to the travel agency, and the reservation process is completed.

[0736] As described above, the present invention provides a system that utilizes information from SNS to enable users to easily automatically generate optimal travel plans.

[0737] The processing flow will be explained below.

[0738] Step 1:

[0739] The server uses the API of a social networking service (SNS) to search for posts based on specific keywords or hashtags (e.g., "travel," "tourist destinations") The data collected through the search includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc.

[0740] Step 2:

[0741] The server stores the collected social media post data in a temporary database, allowing for efficient subsequent analysis. It also stores the metadata of each post for analysis.

[0742] Step 3:

[0743] The server analyzes the collected images using image recognition technology to identify key features (e.g., landmarks, tourist attractions, restaurants), leveraging computer vision techniques to recognize specific landmarks and objects.

[0744] Step 4:

[0745] The server uses generative artificial intelligence techniques to extract relevant text data (e.g., place names, store names, activity details) from post captions and comment sections, and analyzes the extracted text data using natural language processing (NLP) techniques to identify important keywords and phrases.

[0746] Step 5:

[0747] The server statistically analyzes the data extracted through image recognition and text analysis. Specifically, it calculates how frequently each feature appears and evaluates its popularity and relevance. Based on this, the data is classified into categories such as tourist attractions, restaurants, and events.

[0748] Step 6:

[0749] The server creates weekly, monthly, and yearly rankings based on statistically analyzed data. This allows users to easily understand popular spots and activities for each season. This ranking information is also used as reference when creating plans.

[0750] Step 7:

[0751] When a user clicks on an image or piece of information that interests them on the social networking site, the device sends that information to the server, which then collects information on related tourist spots and stores and automatically generates a travel plan based on the user's input.

[0752] Step 8:

[0753] Based on the information the user clicks, the server designs a travel plan that combines tourist spots, restaurants, activities, etc. The generated plan also includes transportation and accommodations, and the schedule is automatically generated taking into account time and budget.

[0754] Step 9:

[0755] The device displays the generated itinerary to the user, who can review it and make adjustments as needed, such as changing the order of the stops or selecting additional tourist attractions.

[0756] Step 10:

[0757] After the user adjusts the plan, the device sends the information back to the server, which then regenerates the plan based on the user's adjustments and sends the final plan to the device.

[0758] Step 11:

[0759] The server sends the confirmed travel plan to the partner travel agency's system via API, allowing the user to purchase and reserve the travel plan directly. The terminal is then redirected to the travel agency's reservation page, where the user can complete payment procedures.

[0760] Example 1

[0761] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0762] Traditional travel planning requires users to manually gather vast amounts of information and create plans, which is time-consuming and labor-intensive. Furthermore, fragmented information on the Internet makes it difficult to efficiently generate comprehensive travel plans. Furthermore, the process of coordinating travel plans and making reservations is cumbersome, placing a significant burden on users.

[0763] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0764] In this invention, the server includes means for automatically collecting information from social networking services on the Internet, means for identifying tourist attractions and stores from the collected information using image recognition technology, means for extracting the identified information as text data using generative artificial intelligence technology, means for statistically analyzing the extracted data and classifying it by category, means for designing a travel plan based on the classified information and automatically generating a schedule that takes time and budget into consideration, means for displaying the generated travel plan on the user's device so that the user can confirm and adjust it, means for automatically generating a new travel plan based on information selected by the user, and means for transmitting the confirmed travel plan to affiliated travel agencies and providing procedures for the user to reserve and purchase the travel plan. This allows users to efficiently generate, adjust, and book an optimal travel plan without any effort.

[0765] A "social networking service" is an online platform that allows users to exchange information and communicate with each other over the Internet.

[0766] "Image recognition technology" is a technology in which a machine analyzes the content of an image and identifies the objects and scenes contained within it.

[0767] "Generative AI technology" is an AI technology that generates natural language sentences, images, etc. based on input data.

[0768] "Text data" refers to data consisting of character information, including sentences and words.

[0769] "Statistical analysis" means analyzing the characteristics and trends of collected data using statistical methods.

[0770] A "travel plan" refers to a plan of the schedule, destinations, activities, etc. for a trip within a specific period of time.

[0771] A "schedule" is a time-based plan that shows the sequence of activities or events that should occur at a particular time.

[0772] "User Device" means a hardware device used by a User, such as a computer, smartphone, or tablet.

[0773] "Travel Agent" means a business or organization that provides travel services, sells travel products, and processes reservations.

[0774] "Reservation procedure" refers to the action of finalizing travel plans and reserving accommodation and transportation based on those plans.

[0775] The present invention is a system that automatically collects information from social networking services (SNS) on the Internet, analyzes the information using image recognition technology and generative artificial intelligence technology, and provides users with optimal travel plans. A detailed description of an embodiment of the present invention will be given below.

[0776] System configuration

[0777] This system mainly consists of a server and a user's device. The server collects information from SNS and analyzes and classifies the data using image recognition and generative artificial intelligence technology. The user's device receives the information, displays the travel plan, and allows adjustments.

[0778] Collecting SNS information

[0779] The server automatically collects posts based on specific keywords and hashtags using the APIs of social media platforms. This includes images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.). For example, it collects information based on hashtags such as "Tokyo travel" and "gourmet."

[0780] Image recognition and text data extraction

[0781] The server applies image recognition technology to the collected images and videos to identify place names, tourist attractions, stores, etc. For example, it automatically extracts famous tourist spots (e.g., Tokyo Tower and Sensoji Temple) and restaurants. It also uses generative artificial intelligence technology to extract additional text data from post captions and comment sections.

[0782] Data classification and analysis

[0783] The server statistically analyzes the extracted data using image recognition and text analysis, categorizing it into categories, such as tourist attractions, restaurants, and events, and generates rankings for each category. These rankings are displayed based on weekly, monthly, and yearly metrics based on the collected data.

[0784] Travel plan design

[0785] When a user clicks on an image or piece of information that interests them, the server automatically generates a travel plan based on that information. The plan includes places to visit, places to eat, activities, etc. For example, if a user clicks on an image of "Tokyo Tower," a schedule is automatically created that takes into account nearby tourist attractions, restaurants, and transportation options.

[0786] Schedule and budget management

[0787] The server generates a detailed schedule (timetable and budget) based on the travel plan and displays it on the user's device. The user can check this schedule and make changes as needed. For example, a detailed plan could be provided, such as visiting Sensoji Temple in the morning, having lunch at a famous okonomiyaki restaurant in Asakusa, and then visiting Tokyo Tower.

[0788] Travel agency collaboration and booking

[0789] The server sends the finalized travel plan to the partner travel agency, allowing the user to purchase and reserve the plan. The user's device is redirected to the travel agency's reservation page, where payment procedures are completed. In this way, the user can enjoy the optimal travel plan without any hassle.

[0790] Specific examples

[0791] For example, a user searches for "Kyoto travel" on a major online social networking site and clicks on a photo that catches their eye. This photo may show a famous tourist spot in Kyoto (e.g., Kiyomizu-dera Temple). Based on this information, the server automatically generates a travel plan that includes Kiyomizu-dera Temple and provides a schedule that takes into account nearby tourist spots, restaurants, and transportation options. The user can also review this plan and select additional tourist spots (e.g., Kinkaku-ji Temple) if necessary. Finally, once the plan is finalized, the server sends this information to the travel agency, and the reservation process is completed.

[0792] Prompt Sentence Examples

[0793] "A user searched for 'Kyoto travel' on a major online social networking site and clicked on a photo that caught their eye (Kiyomizu-dera Temple). Based on this information, please create a Kyoto trip plan that includes Kiyomizu-dera Temple. Please provide a detailed itinerary that takes into account places to visit, restaurants, recommended activities, and transportation options."

[0794] In this way, the present invention provides a system that utilizes information from SNS to enable users to easily automatically generate optimal travel plans.

[0795] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0796] Step 1:

[0797] The server automatically collects social media information. It uses the API of social media platforms, such as the Twitter API, to collect posts based on specific keywords and hashtags (for example, "Tokyo travel" or "food"). The input is the keywords and hashtags specified by the user. The output is the collected images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.). Specifically, the server calls the API to retrieve posts in real time and stores them in a database.

[0798] Step 2:

[0799] The server performs image recognition on the collected images and videos. It uses the Google Cloud Vision API to identify tourist attractions and stores contained in the images. The input is the collected images and videos. The output is information about the identified tourist attractions and stores (e.g., Tokyo Tower, Sensoji Temple). Specifically, the server calls the image recognition API, analyzes objects and text in the images, and stores the results in a database.

[0800] Step 3:

[0801] The server extracts text data using generative artificial intelligence technology. It uses a generative AI model, such as OpenAI's GPT-3.5, to generate text data from the collected post captions and comments. The input is the post captions and comments. The output is the extracted text data. Specifically, the server sends prompts to the generative AI model, analyzes the generated text, and extracts important information.

[0802] Step 4:

[0803] The server statistically analyzes the extracted data and classifies it into categories, such as tourist attractions, restaurants, and events, and generates rankings for each category. The input is information about identified tourist attractions and stores and the extracted text data. The output is the data classified by category and rankings. Specifically, the server retrieves data from the database, applies statistical analysis algorithms to classify the data, and calculates rankings.

[0804] Step 5:

[0805] When a user clicks on an image or piece of information that interests them, the server automatically generates a travel plan based on that information. The input is the images and information selected by the user. The output is an automatically generated travel plan (places to visit, places to eat, activities, etc.). Specifically, the server runs a travel plan generation algorithm based on the selected data to create an optimal schedule for each tourist spot and shop.

[0806] Step 6:

[0807] The server generates a detailed schedule (timetable and budget) based on the travel plan and displays it on the user's device. The input is the automatically generated travel plan. The output is the detailed schedule and budget. Specifically, the server analyzes the travel plan and applies a time and budget management algorithm to determine a specific schedule and sends it to the user's device.

[0808] Step 7:

[0809] The server sends the final confirmed travel plan to the partner travel agency, allowing the user to purchase and book the travel plan. The input is the confirmed travel plan. The output is a notification of completion of transmission to the travel agency and the progress of the reservation. Specifically, the server calls the partner agency's API to send the travel plan and manages the reservation and purchase procedures. On the user's device, the server is redirected to the reservation page and the payment procedure is carried out.

[0810] Through the above steps, this system utilizes information from SNS to enable users to easily automatically generate and book optimal travel plans.

[0811] (Application example 1)

[0812] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0813] Conventional travel planning systems require users to manually search for information on tourist spots, restaurants, and other attractions that interest them and then create a plan based on that information, which is both time-consuming and laborious. Furthermore, generating an optimal travel plan requires integrating a large amount of information, which places a significant burden on users. Furthermore, reservations and payment procedures must be completed separately, which is time-consuming. To solve these issues, a system is needed that automatically collects information that interests users and provides optimal travel plans.

[0814] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0815] In this invention, the server includes means for automatically collecting information from social networking services on the Internet, means for identifying tourist attractions and stores from the collected information using image recognition technology, means for extracting the identified information as text data using generative artificial intelligence technology, means for statistically analyzing the extracted data and classifying it by category, means for designing a travel plan based on the classified information and automatically generating a schedule that takes time and budget into consideration, means for generating an optimal travel plan by allowing the user to select images and information that interest them, means for the user to customize the travel plan proposed by the system, and means for transmitting the final travel plan to affiliated travel agencies and providing reservation and payment procedures. This allows the user to easily generate an optimal travel plan and complete the reservation and payment procedures all at once.

[0816] A "social networking service" is a platform that allows users to communicate with each other over the Internet.

[0817] "Means for automatically collecting information" refers to a system that automatically retrieves related posts from social media based on specific keywords or hashtags.

[0818] "Image recognition technology" is a technology that analyzes image data and identifies specific objects or scenes.

[0819] "Tourist destinations" are areas or places that tourists visit, including historical buildings and natural landscapes.

[0820] A "store" is a facility for selling products and providing services.

[0821] "Generative AI technology" is an AI technology that has the ability to learn patterns from large amounts of data and generate new text and content.

[0822] "Text data" refers to information expressed as a string of characters that can be analyzed and searched.

[0823] "Statistical analysis means" refers to a system that uses statistical methods on collected data to analyze its characteristics and trends.

[0824] A "categorization method" is a mechanism for organizing data into categories according to specific criteria, making them easier to identify.

[0825] A "means for designing travel plans" is a system that plans destinations and activities based on the user's interests and collected information.

[0826] A "means for automatically generating a schedule" is a system that uses a program to automatically create a timetable or itinerary based on a designed travel plan.

[0827] "User device" refers to the terminal used by the user to operate or view the site, including smartphones and personal computers.

[0828] "System-proposed travel plans" refers to travel plans automatically generated by the system.

[0829] "Customizable means" means that users can change or adjust the plans suggested by the system to suit their preferences.

[0830] A "travel agent" is a business or organization that provides travel services and sells and books travel packages to customers.

[0831] A "means for providing reservation and payment procedures" is a mechanism that allows users to easily make reservations and complete payments after finalizing their travel plans.

[0832] This invention is a system that automatically collects social media information, analyzes and classifies it using image recognition technology and generative artificial intelligence technology, and provides optimal travel plans based on that information. This system is mainly composed of a server and user terminals. This system is described in detail below.

[0833] System program configuration

[0834] The server automatically collects information from social media sites. Specifically, it uses the social media site's API to automatically retrieve related posts based on specific keywords and hashtags. The collected information includes images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.).

[0835] The collected image data is analyzed using image recognition technology. The server uses a CLIP (Contrastive Language–Image Pretraining) model to identify tourist attractions and stores from the image. For example, if a famous tourist attraction or restaurant appears in the image, it is automatically extracted.

[0836] Next, generative artificial intelligence techniques are used to extract text data: the server uses models such as GPT-3 to extract additional useful information from the captions and comments sections of social media posts, providing the detailed information users crave.

[0837] The extracted data is statistically analyzed and classified into categories such as tourist attractions, restaurants, and events. Based on this, the server designs a travel plan and automatically generates a schedule that takes time and budget into consideration. When a user selects images and information that interest them, the server generates and suggests a travel plan based on that information.

[0838] The travel plan is displayed on the user's device and can be customized by the user. Based on the information adjusted by the user, the server adjusts and automatically generates the travel plan again.

[0839] The finalized itinerary is sent to the partner travel agency, where users can easily purchase and book the itinerary. Specifically, they are redirected to the travel agency's booking page and can complete the payment online.

[0840] Hardware and software used

[0841] Hardware: High-performance servers (e.g., Amazon EC2)

[0842] software:

[0843] API client: requests (for accessing APIs of social media platforms)

[0844] Image Recognition: Transformers (using CLIP model)

[0845] Generative AI: openai (using GPT-3)

[0846] Specific examples

[0847] For example, a user searches for information about "Tokyo travel" and selects a photo that catches their eye. Let's say the photo shows Tokyo Tower. Based on this information, the server generates a schedule that takes into account surrounding tourist spots, restaurants, and transportation options, including Tokyo Tower. If the user wishes to customize the schedule, the server can also accommodate requests such as adding Ueno Zoo and Shibuya.

[0848] Additionally, example prompts have the following format:

[0849] "Generate a travel plan based on the user's interests. Location: Tokyo Tower, Attractions: Ueno Zoo, Shibuya, Restaurant: High-end sushi restaurant, Activities: Sightseeing, Shopping"

[0850] In this way, the server is a system that automatically generates highly accurate travel plans based on the information obtained and provides them to users.

[0851] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0852] Step 1:

[0853] The server automatically collects information from social media sites based on specific keywords and hashtags. This process uses the social media site's API to retrieve images, videos, text, and metadata (location, number of likes, number of comments, etc.). The input is the keywords and hashtags specified by the user, and the output is the collected social media data.

[0854] Step 2:

[0855] The server extracts image data from the collected SNS data and applies image recognition technology to identify tourist attractions and stores. At this stage, the CLIP model is used to analyze the images. The input is image data obtained from SNS, and the output is information on the identified tourist attractions and stores.

[0856] Step 3:

[0857] Based on the identified information, the server uses generative artificial intelligence technology (GPT-3) to extract useful text data from post captions and comments. The input is the image data and text data of tourist attractions and stores identified in Step 2, and the output is the text data generated by the generative AI.

[0858] Step 4:

[0859] The server statistically analyzes the extracted data and classifies it into categories such as tourist attractions, restaurants, and events. The input is text data and metadata, and the output is information classified by category. The analysis uses major statistical methods.

[0860] Step 5:

[0861] The server designs a travel plan based on the classified information and automatically generates a schedule that takes time and budget into consideration. The input is information categorized by category, and the output is a detailed travel schedule. Specifically, it calculates time allocation and expenses by taking into account multiple destinations and activities.

[0862] Step 6:

[0863] The user can view the generated travel plan on their device and select images and information that interest them. The server then generates the optimal travel plan and presents it to the user. The input is the images and information selected by the user, and the output is a customized travel plan.

[0864] Step 7:

[0865] The user can customize the proposed itinerary through the terminal. The server reflects the user's changes and automatically regenerates the adjusted itinerary. The input is the user's customized information, and the output is the adjusted itinerary.

[0866] Step 8:

[0867] The server then sends the finalized itinerary to the affiliated travel agency. The user then completes the procedure to purchase and reserve the itinerary via the terminal. The input is the finalized itinerary, and the output is the data to be sent to the travel agency and a notification of reservation completion.

[0868] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0869] This invention relates to a system that automatically collects information from online social networking services (SNS) and analyzes it using image recognition and generative artificial intelligence technologies to provide users with optimal travel plans. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, it provides users with a more personalized travel experience.

[0870] System configuration

[0871] This system mainly consists of a server, a user's device, and an emotion engine. The server collects information from SNS and analyzes and classifies the data using image recognition and generative artificial intelligence technology. The user's device receives the information, displays the travel plan, and allows for adjustments. The emotion engine also identifies the user's emotions and sends the data to the server.

[0872] Collection and analysis of SNS information

[0873] The server automatically collects posts based on specific keywords and hashtags using the APIs of social media platforms, including images, videos, posting dates and times, poster information, location information, hashtags, number of likes, and number of comments.

[0874] The server applies image recognition technology to the collected images and videos to identify place names, tourist attractions, stores, etc. It also uses generative artificial intelligence technology to extract relevant text data from post captions and comment sections, resulting in information categorized into tourist attractions, restaurants, events, etc.

[0875] Emotion Engine Functions

[0876] The user's device is equipped with a camera and microphone to recognize the user's emotions in real time. The emotion engine analyzes these sensor data and identifies the user's emotions (e.g., joy, surprise, excitement) from facial expressions and tone of voice. The emotion data recognized by the emotion engine is sent to a server for analysis.

[0877] Travel plan design and coordination

[0878] When a user clicks on an image or piece of information that interests them, the device sends that information to the server. Based on this, the server collects information on related tourist spots and stores, and automatically generates a travel plan that takes into account the user's preferences and emotional data. The generated plan also includes transportation and places to stay, and a schedule that takes time and budget into consideration is automatically created.

[0879] For example, if a user clicks on an image of a "trip to Paris" and expresses joy, the server will use that information to suggest a plan that combines popular tourist spots such as the Eiffel Tower and the Louvre.

[0880] View and rearrange your plan

[0881] The device displays the generated itinerary to the user, who can review it and make adjustments as needed, such as changing the order of the stops or selecting additional tourist attractions.

[0882] After the user adjusts the plan, the device sends the information back to the server, which then regenerates the plan based on the user's adjustments and the latest emotion data, and sends the final plan to the device.

[0883] Travel agency collaboration and booking

[0884] The server sends the finalized travel plan to the partner travel agency's system via API, allowing the user to purchase and reserve the travel plan directly. The terminal is then redirected to the travel agency's reservation page, where the user can complete payment procedures, etc. This allows the user to enjoy the optimal travel plan without any hassle.

[0885] As described above, the present invention provides a system that allows users to easily automatically generate optimal travel plans by utilizing information from SNS and combining it with user emotional data.

[0886] The processing flow will be explained below.

[0887] Step 1:

[0888] The server uses the API of a social networking service (SNS) to search for posts based on specific keywords or hashtags (e.g., "travel," "tourist destinations") The data collected through the search includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc.

[0889] Step 2:

[0890] The server stores the collected social media post data in a temporary database, allowing for efficient subsequent analysis. It also stores the metadata of each post for analysis.

[0891] Step 3:

[0892] The server analyzes the collected images using image recognition technology to identify key features (e.g., landmarks, tourist attractions, restaurants), leveraging computer vision techniques to recognize specific landmarks and objects.

[0893] Step 4:

[0894] The server uses generative artificial intelligence techniques to extract relevant text data (e.g., place names, store names, activity details) from post captions and comment sections, and analyzes the extracted text data using natural language processing (NLP) techniques to identify important keywords and phrases.

[0895] Step 5:

[0896] The server statistically analyzes the data extracted through image recognition and text analysis. Specifically, it calculates how frequently each feature appears and evaluates its popularity and relevance. Based on this, the data is classified into categories such as tourist attractions, restaurants, and events.

[0897] Step 6:

[0898] The server creates weekly, monthly, and yearly rankings based on statistically analyzed data. This allows users to easily understand popular spots and activities for each season. This ranking information is also used as reference when creating plans.

[0899] Step 7:

[0900] The camera and microphone on the user's device capture the user's facial expressions and tone of voice, which are then analyzed in real time by the emotion engine.

[0901] Step 8:

[0902] The emotion engine analyzes the user's emotion data (e.g., joy, surprise, excitement) and sends it to the server, where it is linked to the images and information the user is viewing.

[0903] Step 9:

[0904] When a user clicks on an SNS image or information that interests them, the device sends that information and linked emotional data to the server. Based on this, the server collects information on related tourist spots and stores, and automatically generates a travel plan taking into account the user's preference information and emotional data.

[0905] Step 10:

[0906] The server then designs a travel plan based on the user's emotional data, including activities that emphasize fun, based on a strong sense of "joy." The plan also includes transportation and accommodations, and automatically generates a schedule that takes into account time and budget.

[0907] Step 11:

[0908] The device displays the generated itinerary to the user, who can review it and make adjustments as needed, such as changing the order of the stops or selecting additional tourist attractions.

[0909] Step 12:

[0910] After the user adjusts the plan, the device sends the information back to the server, which then regenerates the plan based on the user's adjustments and the latest emotion data, and sends the final plan to the device.

[0911] Step 13:

[0912] The server sends the confirmed travel plan to the partner travel agency's system via API, allowing the user to purchase and reserve the travel plan directly. The terminal is then redirected to the travel agency's reservation page, where the user can complete payment procedures.

[0913] Example 2

[0914] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0915] Conventional travel plan creation systems have difficulty automatically generating personalized plans that reflect the user's emotions and individual preferences. Furthermore, there is a lack of a method for analyzing user emotions in real time and reflecting them in travel plans, making it difficult to improve the quality of the user experience.

[0916] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0917] In this invention, the server includes means for automatically collecting information from social media services on the Internet, means for identifying tourist attractions and stores from the collected information using image recognition technology, means for extracting the identified information as text data using generative artificial intelligence technology, means for collecting and analyzing emotional data in real time using an engine that recognizes user emotions, means for statistically analyzing the extracted data and the collected emotional data and classifying them by category, means for designing a travel plan based on the classified information and analyzed emotional data and automatically generating a schedule that takes time and budget into consideration, and means for displaying the generated travel plan on the user's device so that the user can check and adjust it. This enables the automatic generation of a personalized travel plan that takes into consideration the user's individual preferences and emotions.

[0918] "Internet social media service" refers to an online platform that enables users to share information and interact with others via the Internet.

[0919] "Image recognition technology" refers to the process of automatically identifying and identifying objects and text within an image using computer vision technology.

[0920] "Generative AI technology" refers to AI technology that has the ability to learn patterns from large amounts of data and generate new data and information.

[0921] "User emotion recognition engine" refers to technology that identifies emotions by analyzing a user's facial expressions and tone of voice.

[0922] "Statistical analysis" refers to the process of quantifying large amounts of data and analyzing its patterns and correlations using statistical methods.

[0923] "Categorizing" refers to the process of grouping collected data based on specific attributes or themes.

[0924] "Designing a travel plan" refers to the process of creating a travel schedule and itinerary by combining tourist attractions, activities, means of transportation, etc.

[0925] "Automatically generating a schedule that takes time and budget into consideration" refers to the process of automatically creating an optimal travel schedule using a computer based on the user's desired time period and budget restrictions.

[0926] "Displaying it on the user's device and allowing the user to check and adjust it" means displaying the generated travel plan on a device such as a smartphone or tablet, allowing the user to check and change its contents.

[0927] The present invention is a system that automatically collects information from social media services on the Internet and analyzes that information using image recognition technology and generative artificial intelligence technology to provide users with optimal travel plans. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system provides a more personalized travel experience. Specific embodiments of the system are described below.

[0928] System configuration

[0929] This system consists of a server, a user's terminal, and an emotion engine.

[0930] Collecting SNS information

[0931] The server automatically collects posts based on specific keywords or hashtags using the APIs of social media platforms, such as Twitter API or Instagram Graph API. The collected data includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, and number of comments.

[0932] Image Recognition and Text Analytics

[0933] The server applies image recognition technology (e.g., Google Cloud Vision API) to the collected images and videos to identify place names, tourist attractions, stores, etc. It then uses generative artificial intelligence technology (e.g., OpenAI's GPT-4) to analyze posted captions and comments and extract related text data. This allows it to obtain information categorized into tourist attractions, restaurants, events, etc.

[0934] Emotion Engine Functions

[0935] The user's device is equipped with a camera and microphone, which the emotion engine uses to identify emotions in real time from the user's facial expressions and tone of voice. The emotion engine uses, for example, Microsoft Azure's Emotion API. The collected emotion data is sent to a server and used to plan the itinerary.

[0936] Travel plan design

[0937] When a user clicks on an image or piece of information that interests them, the server uses that information to collect information on related tourist spots and stores. Taking into account the user's emotional data and preference information, the server automatically generates a travel plan. The generated plan includes a schedule that takes into account transportation, places to stay, time, and budget. For example, if a user clicks on an image of a "trip to Paris" and expresses joy, the server will suggest a plan that includes popular tourist spots such as the Eiffel Tower, the Louvre, and the Champs-Élysées.

[0938] View and adjust your plan

[0939] The device displays the generated travel plan to the user. The user can review the plan, change the order of the destinations, or select additional tourist attractions. The adjusted plan information is sent back to the server, and the server regenerates the plan based on the user's adjustments and the latest emotion data, and sends the final plan to the device.

[0940] Book a travel plan

[0941] The server sends the finalized itinerary to the partner travel agency's system via API, and the terminal redirects the user to the travel agency's booking page. The user can then proceed with the procedure to purchase and reserve the itinerary, for example, using Expedia API or Booking.com API.

[0942] Examples and prompts

[0943] For example, "If a user clicks on an image of a trip to Paris and expresses joy, the server will suggest a travel itinerary that includes attractions such as the Eiffel Tower, the Louvre, and the Champs-Élysées."

[0944] An example of a prompt for a generative AI model is, "What is the best plan for a trip to Paris? Please list it in categories of tourist attractions, restaurants, and events."

[0945] The above is a detailed description of the mode for carrying out the invention.

[0946] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0947] Step 1: Collect social media information

[0948] Input: Specific keywords or hashtags

[0949] Specific operation: The server uses the API of the social media platform (e.g., Twitter API, Instagram Graph API) to collect posts that match the specified keywords or hashtags.

[0950] Output: Images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc.

[0951] Step 2: Image Recognition

[0952] Input: Collected image and video data

[0953] Specific operation: The server uses image recognition technology (e.g., Google Cloud Vision API) to identify place names, tourist attractions, stores, etc. from this data.

[0954] Output: Information on identified tourist attractions and stores, and corresponding image and video data

[0955] Step 3: Text analysis

[0956] Input: Collected post captions and comments

[0957] How it works: The server uses generative artificial intelligence techniques (e.g., OpenAI's GPT-4) to extract relevant text data from captions and comments and classify them into categories such as tourist attractions, restaurants, and events.

[0958] Output: Extracted text data and categorical information

[0959] Step 4: Collecting emotion data

[0960] Input: User camera and microphone data

[0961] How it works: The device uses an emotion engine (e.g., Microsoft Azure Emotion API) to identify emotions in real time from the user's facial expressions and tone of voice.

[0962] Output: Collected emotion data, identified emotion types

[0963] Step 5: Analyze the sentiment data

[0964] Input: Collected emotion data

[0965] Specific operation: The server analyzes the collected emotional data to identify the user's current emotional state. Based on this analysis result, it is used to generate a travel plan.

[0966] Output: Parsed emotion data

[0967] Step 6: Generate your travel plan

[0968] Input: Identified tourist destination information, extracted text data, analyzed emotion data, user preference information

[0969] Specific operation: Based on this input data, the server collects information on tourist spots and stores, and automatically generates a travel plan taking into account the user's emotional data and preference information.

[0970] Output: Automatically generated travel plan, schedule information, transportation, and accommodation

[0971] Step 7: View your travel plans

[0972] Input: Auto-generated itinerary

[0973] Specific operation: The terminal displays the generated travel plan to the user so that the user can confirm it.

[0974] Output: Travel plan displayed on the user's device

[0975] Step 8: Adjust your travel plans

[0976] Input: User adjustment instructions

[0977] Specific operation: The user checks the displayed plan, changes the order of the stops, or selects additional tourist attractions. The adjustments are sent to the server via the device.

[0978] Output: Adjusted travel plan instructions

[0979] Step 9: Regenerate the adjusted plan

[0980] Input: Adjusted travel plan instructions, latest sentiment data

[0981] Specific operation: The server generates a new travel plan based on these input data and sends the final plan to the terminal.

[0982] Output: Final itinerary

[0983] Step 10: Book your travel plan

[0984] Input: Final itinerary

[0985] Specific operation: The server sends the finalized travel plan to the partner travel agency's system via API, and the terminal redirects the user to the travel agency's booking page, where the user completes the booking procedure.

[0986] Output: Confirmed reservation information, redirect to reservation completion page

[0987] The above is a detailed description of the specific steps of the program processing of this system.

[0988] (Application example 2)

[0989] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0990] Conventional travel plan providing systems have the problem of being unable to flexibly respond to users' emotions and individual needs. They also lack a means to effectively communicate travel plans to users in an easy-to-understand manner. As a result, users end up spending a lot of time and effort selecting travel destinations and planning their schedules.

[0991] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for automatically collecting information from social networking services on the Internet, means for identifying tourist spots and stores from the collected information using image recognition technology, and means for extracting the identified information as text data using generative artificial intelligence technology. This makes it possible to recognize the user's emotions and personalize the travel plan using the emotion data. Furthermore, by adding means for generating a travel plan based on emotions in video format, the travel plan can be communicated to the user visually and effectively.

[0992] An "online social networking service" is a platform that allows users to share information and communicate with each other via the Internet.

[0993] "Means of automatically collecting information" refers to a method of automatically obtaining posted data from social networking services on the Internet using specific keywords or hashtags through a program.

[0994] "Image recognition technology" is a technology in which a computer analyzes image data and identifies specific objects or landmarks.

[0995] "Generative artificial intelligence technology" is an artificial intelligence technology that analyzes collected data and automatically generates text and data.

[0996] "Means of extracting as text data" refers to a method of extracting relevant text information from collected data using image recognition technology or generative artificial intelligence technology and providing it in a meaningful format.

[0997] "Means of statistical analysis and categorization" refers to a method of analyzing extracted data using statistical analysis and dividing it into categories such as tourist attractions, stores, and events based on their relevance.

[0998] "A means for designing travel plans and automatically generating schedules that take time and budget into consideration" is a method for creating travel plans based on classified information and automatically generating the optimal schedule to meet the user's requirements.

[0999] "Means for displaying on the user's device and allowing the user to review and adjust" refers to a method for displaying the generated travel plan on the user's smartphone or computer, allowing the user to review the plan and make changes as necessary.

[1000] "Means for recognizing a user's emotions and using that emotional data to personalize travel plans" refers to a method for identifying a user's emotions by analyzing their facial expressions and tone of voice, and then using that data to provide travel plans that match the user's preferences.

[1001] The "means for generating emotion-based travel plans in video format" is a method for generating videos and providing them to users based on information about tourist spots and events that take into account the user's emotional data.

[1002] The present invention relates to a system that collects information from social networking services (SNS) and provides travel plans in the form of videos that take into account the user's emotions. Specific embodiments of the system will be described below.

[1003] This system mainly consists of a server, a user's device, and an emotion engine. The server includes a social networking data collection module, a data analysis module, a travel plan generation module, and a video generation module. The user's device is equipped with a camera and microphone and includes an emotion engine for emotion recognition.

[1004] Program Overview

[1005] 1. Collecting social media data

[1006] The server uses APIs from social media platforms to automatically collect data based on specific keywords and hashtags, including images, videos, posting dates and times, poster information, location information, hashtags, number of likes, number of comments, etc. Specific software used includes social media APIs and data collection libraries.

[1007] 2. Data Analysis

[1008] The collected data is then analyzed using image recognition technology (e.g., OpenCV and TensorFlow) to identify tourist attractions and stores. Generative AI technology is then used to extract relevant text data from post captions and comment sections, allowing each tourist attraction and store to be classified into a category.

[1009] 3. Emotional Recognition

[1010] The user's device is equipped with a camera and microphone, and by analyzing these sensor data, the emotion engine recognizes the user's emotions in real time. The emotion engine identifies whether the user is expressing emotions such as joy or surprise from facial expressions and tone of voice. This data is sent to a server and used to customize the travel plan. Specific software includes emotion recognition libraries (e.g., FaceReader and Vokaturi).

[1011] 4. Designing a travel plan

[1012] The server designs a travel plan based on the information the user clicks on and their emotional data. The plan includes sightseeing spots, shops, events, etc., and the schedule is automatically generated taking into account time and budget. The generated plan can be adjusted to suit the user's preferences.

[1013] 5. Video Generation and Display

[1014] The finalized itinerary is then generated in video format by a video generation module. The generated video is sent to the user's device, where it presents the itinerary in a visually easy-to-understand format. Specific software includes a video editing library (e.g., FFmpeg, MoviePy, etc.).

[1015] Specific examples

[1016] For example, if a user expresses interest in a trip to Paris, the system collects data related to Paris from social media and analyzes information on tourist attractions and restaurants. If the user clicks on an image of Paris on the screen and expresses joy, the server will suggest a travel plan that combines popular spots such as the Eiffel Tower, the Louvre, and Montmartre. A video based on the plan is generated and provided to the user.

[1017] Prompt Sentence Examples

[1018] "If the user's analyzed emotion is detected as 'surprise,' generate a video of 'recommended travel destinations full of surprises.' For example, suggest a plan that includes unique experiences such as an underwater hotel or a trapeze."

[1019] As described above, the system of the present invention can provide users with optimal travel plans in video format by combining information collected from SNS with user emotion data.

[1020] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1021] Step 1:

[1022] The server automatically collects data from social networking services on the Internet using keywords and hashtags. Keywords that reflect the user's interests (e.g., "trip to Paris") are used for this input. The collected data includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc. The specific software used to collect the data is the SNS API or data collection library. The server obtains this posted data.

[1023] Step 2:

[1024] The server uses image recognition technology (such as OpenCV or TensorFlow) on the collected data to identify tourist attractions and stores. The input is the image data collected in step 1, and the output is text data in which tourist attractions and stores are identified. Image recognition technology is used to identify tourist attraction landmarks, store logos, etc., and the identified information is extracted as text.

[1025] Step 3:

[1026] The server uses generative AI technology to extract relevant text data from post captions and comment sections. The input is the text data of the social media post, and the output is the analyzed text data. Generative AI technology is used to understand the content of the post and extract information about tourist spots, stores, and events. Specifically, a natural language processing library is used.

[1027] Step 4:

[1028] The server statistically analyzes the extracted text data and classifies it into categories such as tourist attractions, stores, and events. The input is the text data extracted in step 3, and the output is information classified by category. The server analyzes the data using a statistical analysis library and classifies it into each category.

[1029] Step 5:

[1030] The server uses the user's emotional data to design a travel plan and automatically generate a schedule that takes time and budget into consideration. The input is the information classified in step 4 and the user's emotional data. The emotion engine uses the user's camera and microphone to obtain emotional data in real time (e.g., joy, surprise, etc.) and generates a travel plan based on those emotions. Specifically, the emotion engine uses FaceReader and Vokaturi.

[1031] Step 6:

[1032] The server generates a travel plan based on emotions in video format and sends it to the user's device. The input is the travel plan generated in step 5, and the output is a video travel plan. Using a video generation module (e.g., FFmpeg or MoviePy), videos of tourist spots and events according to emotions are created and visually presented to the user.

[1033] Step 7:

[1034] The device displays the generated video to the user, who then reviews the plan and adjusts it as necessary. The input is the video generated in step 6, and the output is the travel plan adjusted by the user. The user can review the travel plan through the device and make adjustments, such as changing the order of the destinations. The adjusted information is sent back to the server, and the plan is regenerated as necessary.

[1035] The above is a description of the specific processing steps of the system that realizes the application example.

[1036] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1037] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1038] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1039] [Fourth embodiment]

[1040] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1041] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1042] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1043] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1044] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1045] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1046] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1047] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1048] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1049] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1050] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1051] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1052] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1053] The present invention is a system that automatically collects information from social networking services (SNS) on the Internet, analyzes the information using image recognition technology and generative artificial intelligence technology, and provides users with optimal travel plans. The following describes in detail the embodiments of the present invention.

[1054] System configuration

[1055] This system mainly consists of a server and a user's device. The server collects information from SNS and analyzes and classifies the data using image recognition and generative artificial intelligence technology. The user's device receives the information, displays the travel plan, and allows adjustments.

[1056] Collecting SNS information

[1057] The server automatically collects posts based on specific keywords and hashtags using the APIs of social media platforms. This includes images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.). For example, it collects information based on hashtags such as "Tokyo travel" and "gourmet."

[1058] Image recognition and text data extraction

[1059] The server applies image recognition technology to the collected images and videos to identify place names, tourist attractions, stores, etc. For example, it automatically extracts famous tourist spots (Tokyo Tower and Sensoji Temple) and restaurants. It also uses generative artificial intelligence technology to extract additional text data from post captions and comment sections.

[1060] Data classification and analysis

[1061] The server statistically analyzes the extracted data using image recognition and text analysis, categorizing it into categories, such as tourist attractions, restaurants, and events, and generates rankings for each category. These rankings are displayed based on weekly, monthly, and yearly metrics based on the collected data.

[1062] Travel plan design

[1063] When a user clicks on an image or piece of information that interests them, the server automatically generates a travel plan based on that information. The plan includes places to visit, places to eat, activities, etc. For example, if a user clicks on an image of "Tokyo Tower," a schedule is automatically created that takes into account nearby tourist attractions, restaurants, and transportation options.

[1064] Schedule and budget management

[1065] The server generates a detailed schedule (timetable and budget) based on the travel plan and displays it on the user's device. The user can check this schedule and make changes as needed. For example, a detailed plan could be provided, such as visiting Sensoji Temple in the morning, having lunch at a famous okonomiyaki restaurant in Asakusa, and then visiting Tokyo Tower.

[1066] Travel agency collaboration and booking

[1067] The server sends the finalized travel plan to the partner travel agency, allowing the user to purchase and reserve the plan. The user's device is redirected to the travel agency's reservation page, where payment procedures are completed. In this way, the user can enjoy the optimal travel plan without any hassle.

[1068] Specific examples

[1069] For example, a user searches for "Kyoto travel" on a major online social networking site and clicks on a photo that catches their eye. This photo may show a famous tourist spot in Kyoto (e.g., Kiyomizu-dera Temple). Based on this information, the server automatically generates a travel plan that includes Kiyomizu-dera Temple and provides a schedule that takes into account nearby tourist spots, restaurants, and transportation options. The user can also review this plan and select additional tourist spots (e.g., Kinkaku-ji Temple) if necessary. Finally, once the plan is finalized, the server sends this information to the travel agency, and the reservation process is completed.

[1070] As described above, the present invention provides a system that utilizes information from SNS to enable users to easily automatically generate optimal travel plans.

[1071] The processing flow will be explained below.

[1072] Step 1:

[1073] The server uses the API of a social networking service (SNS) to search for posts based on specific keywords or hashtags (e.g., "travel," "tourist destinations") The data collected through the search includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc.

[1074] Step 2:

[1075] The server stores the collected social media post data in a temporary database, allowing for efficient subsequent analysis. It also stores the metadata of each post for analysis.

[1076] Step 3:

[1077] The server analyzes the collected images using image recognition technology to identify key features (e.g., landmarks, tourist attractions, restaurants), leveraging computer vision techniques to recognize specific landmarks and objects.

[1078] Step 4:

[1079] The server uses generative artificial intelligence techniques to extract relevant text data (e.g., place names, store names, activity details) from post captions and comment sections, and analyzes the extracted text data using natural language processing (NLP) techniques to identify important keywords and phrases.

[1080] Step 5:

[1081] The server statistically analyzes the data extracted through image recognition and text analysis. Specifically, it calculates how frequently each feature appears and evaluates its popularity and relevance. Based on this, the data is classified into categories such as tourist attractions, restaurants, and events.

[1082] Step 6:

[1083] The server creates weekly, monthly, and yearly rankings based on statistically analyzed data. This allows users to easily understand popular spots and activities for each season. This ranking information is also used as reference when creating plans.

[1084] Step 7:

[1085] When a user clicks on an image or piece of information that interests them on the social networking site, the device sends that information to the server, which then collects information on related tourist spots and stores and automatically generates a travel plan based on the user's input.

[1086] Step 8:

[1087] Based on the information the user clicks, the server designs a travel plan that combines tourist spots, restaurants, activities, etc. The generated plan also includes transportation and accommodations, and the schedule is automatically generated taking into account time and budget.

[1088] Step 9:

[1089] The device displays the generated itinerary to the user, who can review it and make adjustments as needed, such as changing the order of the stops or selecting additional tourist attractions.

[1090] Step 10:

[1091] After the user adjusts the plan, the device sends the information back to the server, which then regenerates the plan based on the user's adjustments and sends the final plan to the device.

[1092] Step 11:

[1093] The server sends the confirmed travel plan to the partner travel agency's system via API, allowing the user to purchase and reserve the travel plan directly. The terminal is then redirected to the travel agency's reservation page, where the user can complete payment procedures.

[1094] Example 1

[1095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1096] Traditional travel planning requires users to manually gather vast amounts of information and create plans, which is time-consuming and labor-intensive. Furthermore, fragmented information on the Internet makes it difficult to efficiently generate comprehensive travel plans. Furthermore, the process of coordinating travel plans and making reservations is cumbersome, placing a significant burden on users.

[1097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1098] In this invention, the server includes means for automatically collecting information from social networking services on the Internet, means for identifying tourist attractions and stores from the collected information using image recognition technology, means for extracting the identified information as text data using generative artificial intelligence technology, means for statistically analyzing the extracted data and classifying it by category, means for designing a travel plan based on the classified information and automatically generating a schedule that takes time and budget into consideration, means for displaying the generated travel plan on the user's device so that the user can confirm and adjust it, means for automatically generating a new travel plan based on information selected by the user, and means for transmitting the confirmed travel plan to affiliated travel agencies and providing procedures for the user to reserve and purchase the travel plan. This allows users to efficiently generate, adjust, and book an optimal travel plan without any effort.

[1099] A "social networking service" is an online platform that allows users to exchange information and communicate with each other over the Internet.

[1100] "Image recognition technology" is a technology in which a machine analyzes the content of an image and identifies the objects and scenes contained within it.

[1101] "Generative AI technology" is an AI technology that generates natural language sentences, images, etc. based on input data.

[1102] "Text data" refers to data consisting of character information, including sentences and words.

[1103] "Statistical analysis" means analyzing the characteristics and trends of collected data using statistical methods.

[1104] A "travel plan" refers to a plan of the schedule, destinations, activities, etc. for a trip within a specific period of time.

[1105] A "schedule" is a time-based plan that shows the sequence of activities or events that should occur at a particular time.

[1106] "User Device" means a hardware device used by a User, such as a computer, smartphone, or tablet.

[1107] "Travel Agent" means a business or organization that provides travel services, sells travel products, and processes reservations.

[1108] "Reservation procedure" refers to the action of finalizing travel plans and reserving accommodation and transportation based on those plans.

[1109] The present invention is a system that automatically collects information from social networking services (SNS) on the Internet, analyzes the information using image recognition technology and generative artificial intelligence technology, and provides users with optimal travel plans. A detailed description of an embodiment of the present invention will be given below.

[1110] System configuration

[1111] This system mainly consists of a server and a user's device. The server collects information from SNS and analyzes and classifies the data using image recognition and generative artificial intelligence technology. The user's device receives the information, displays the travel plan, and allows adjustments.

[1112] Collecting SNS information

[1113] The server automatically collects posts based on specific keywords and hashtags using the APIs of social media platforms. This includes images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.). For example, it collects information based on hashtags such as "Tokyo travel" and "gourmet."

[1114] Image recognition and text data extraction

[1115] The server applies image recognition technology to the collected images and videos to identify place names, tourist attractions, stores, etc. For example, it automatically extracts famous tourist spots (e.g., Tokyo Tower and Sensoji Temple) and restaurants. It also uses generative artificial intelligence technology to extract additional text data from post captions and comment sections.

[1116] Data classification and analysis

[1117] The server statistically analyzes the extracted data using image recognition and text analysis, categorizing it into categories, such as tourist attractions, restaurants, and events, and generates rankings for each category. These rankings are displayed based on weekly, monthly, and yearly metrics based on the collected data.

[1118] Travel plan design

[1119] When a user clicks on an image or piece of information that interests them, the server automatically generates a travel plan based on that information. The plan includes places to visit, places to eat, activities, etc. For example, if a user clicks on an image of "Tokyo Tower," a schedule is automatically created that takes into account nearby tourist attractions, restaurants, and transportation options.

[1120] Schedule and budget management

[1121] The server generates a detailed schedule (timetable and budget) based on the travel plan and displays it on the user's device. The user can check this schedule and make changes as needed. For example, a detailed plan could be provided, such as visiting Sensoji Temple in the morning, having lunch at a famous okonomiyaki restaurant in Asakusa, and then visiting Tokyo Tower.

[1122] Travel agency collaboration and booking

[1123] The server sends the finalized travel plan to the partner travel agency, allowing the user to purchase and reserve the plan. The user's device is redirected to the travel agency's reservation page, where payment procedures are completed. In this way, the user can enjoy the optimal travel plan without any hassle.

[1124] Specific examples

[1125] For example, a user searches for "Kyoto travel" on a major online social networking site and clicks on a photo that catches their eye. This photo may show a famous tourist spot in Kyoto (e.g., Kiyomizu-dera Temple). Based on this information, the server automatically generates a travel plan that includes Kiyomizu-dera Temple and provides a schedule that takes into account nearby tourist spots, restaurants, and transportation options. The user can also review this plan and select additional tourist spots (e.g., Kinkaku-ji Temple) if necessary. Finally, once the plan is finalized, the server sends this information to the travel agency, and the reservation process is completed.

[1126] Prompt Sentence Examples

[1127] "A user searched for 'Kyoto travel' on a major online social networking site and clicked on a photo that caught their eye (Kiyomizu-dera Temple). Based on this information, please create a Kyoto trip plan that includes Kiyomizu-dera Temple. Please provide a detailed itinerary that takes into account places to visit, restaurants, recommended activities, and transportation options."

[1128] In this way, the present invention provides a system that utilizes information from SNS to enable users to easily automatically generate optimal travel plans.

[1129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1130] Step 1:

[1131] The server automatically collects social media information. It uses the API of social media platforms, such as the Twitter API, to collect posts based on specific keywords and hashtags (for example, "Tokyo travel" or "food"). The input is the keywords and hashtags specified by the user. The output is the collected images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.). Specifically, the server calls the API to retrieve posts in real time and stores them in a database.

[1132] Step 2:

[1133] The server performs image recognition on the collected images and videos. It uses the Google Cloud Vision API to identify tourist attractions and stores contained in the images. The input is the collected images and videos. The output is information about the identified tourist attractions and stores (e.g., Tokyo Tower, Sensoji Temple). Specifically, the server calls the image recognition API, analyzes objects and text in the images, and stores the results in a database.

[1134] Step 3:

[1135] The server extracts text data using generative artificial intelligence technology. It uses a generative AI model, such as OpenAI's GPT-3.5, to generate text data from the collected post captions and comments. The input is the post captions and comments. The output is the extracted text data. Specifically, the server sends prompts to the generative AI model, analyzes the generated text, and extracts important information.

[1136] Step 4:

[1137] The server statistically analyzes the extracted data and classifies it into categories, such as tourist attractions, restaurants, and events, and generates rankings for each category. The input is information about identified tourist attractions and stores and the extracted text data. The output is the data classified by category and rankings. Specifically, the server retrieves data from the database, applies statistical analysis algorithms to classify the data, and calculates rankings.

[1138] Step 5:

[1139] When a user clicks on an image or piece of information that interests them, the server automatically generates a travel plan based on that information. The input is the images and information selected by the user. The output is an automatically generated travel plan (places to visit, places to eat, activities, etc.). Specifically, the server runs a travel plan generation algorithm based on the selected data to create an optimal schedule for each tourist spot and shop.

[1140] Step 6:

[1141] The server generates a detailed schedule (timetable and budget) based on the travel plan and displays it on the user's device. The input is the automatically generated travel plan. The output is the detailed schedule and budget. Specifically, the server analyzes the travel plan and applies a time and budget management algorithm to determine a specific schedule and sends it to the user's device.

[1142] Step 7:

[1143] The server sends the final confirmed travel plan to the partner travel agency, allowing the user to purchase and book the travel plan. The input is the confirmed travel plan. The output is a notification of completion of transmission to the travel agency and the progress of the reservation. Specifically, the server calls the partner agency's API to send the travel plan and manages the reservation and purchase procedures. On the user's device, the server is redirected to the reservation page and the payment procedure is carried out.

[1144] Through the above steps, this system utilizes information from SNS to enable users to easily automatically generate and book optimal travel plans.

[1145] (Application example 1)

[1146] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1147] Conventional travel planning systems require users to manually search for information on tourist spots, restaurants, and other attractions that interest them and then create a plan based on that information, which is both time-consuming and laborious. Furthermore, generating an optimal travel plan requires integrating a large amount of information, which places a significant burden on users. Furthermore, reservations and payment procedures must be completed separately, which is time-consuming. To solve these issues, a system is needed that automatically collects information that interests users and provides optimal travel plans.

[1148] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1149] In this invention, the server includes means for automatically collecting information from social networking services on the Internet, means for identifying tourist attractions and stores from the collected information using image recognition technology, means for extracting the identified information as text data using generative artificial intelligence technology, means for statistically analyzing the extracted data and classifying it by category, means for designing a travel plan based on the classified information and automatically generating a schedule that takes time and budget into consideration, means for generating an optimal travel plan by allowing the user to select images and information that interest them, means for the user to customize the travel plan proposed by the system, and means for transmitting the final travel plan to affiliated travel agencies and providing reservation and payment procedures. This allows the user to easily generate an optimal travel plan and complete the reservation and payment procedures all at once.

[1150] A "social networking service" is a platform that allows users to communicate with each other over the Internet.

[1151] "Means for automatically collecting information" refers to a system that automatically retrieves related posts from social media based on specific keywords or hashtags.

[1152] "Image recognition technology" is a technology that analyzes image data and identifies specific objects or scenes.

[1153] "Tourist destinations" are areas or places that tourists visit, including historical buildings and natural landscapes.

[1154] A "store" is a facility for selling products and providing services.

[1155] "Generative AI technology" is an AI technology that has the ability to learn patterns from large amounts of data and generate new text and content.

[1156] "Text data" refers to information expressed as a string of characters that can be analyzed and searched.

[1157] "Statistical analysis means" refers to a system that uses statistical methods on collected data to analyze its characteristics and trends.

[1158] A "categorization method" is a mechanism for organizing data into categories according to specific criteria, making them easier to identify.

[1159] A "means for designing travel plans" is a system that plans destinations and activities based on the user's interests and collected information.

[1160] A "means for automatically generating a schedule" is a system that uses a program to automatically create a timetable or itinerary based on a designed travel plan.

[1161] "User device" refers to the terminal used by the user to operate or view the site, including smartphones and personal computers.

[1162] "System-proposed travel plans" refers to travel plans automatically generated by the system.

[1163] "Customizable means" means that users can change or adjust the plans suggested by the system to suit their preferences.

[1164] A "travel agent" is a business or organization that provides travel services and sells and books travel packages to customers.

[1165] A "means for providing reservation and payment procedures" is a mechanism that allows users to easily make reservations and complete payments after finalizing their travel plans.

[1166] This invention is a system that automatically collects social media information, analyzes and classifies it using image recognition technology and generative artificial intelligence technology, and provides optimal travel plans based on that information. This system is mainly composed of a server and user terminals. This system is described in detail below.

[1167] System program configuration

[1168] The server automatically collects information from social media sites. Specifically, it uses the social media site's API to automatically retrieve related posts based on specific keywords and hashtags. The collected information includes images, videos, and metadata (posting date and time, location information, hashtags, number of likes, number of comments, etc.).

[1169] The collected image data is analyzed using image recognition technology. The server uses a CLIP (Contrastive Language–Image Pretraining) model to identify tourist attractions and stores from the image. For example, if a famous tourist attraction or restaurant appears in the image, it is automatically extracted.

[1170] Next, generative artificial intelligence techniques are used to extract text data: the server uses models such as GPT-3 to extract additional useful information from the captions and comments sections of social media posts, providing the detailed information users crave.

[1171] The extracted data is statistically analyzed and classified into categories such as tourist attractions, restaurants, and events. Based on this, the server designs a travel plan and automatically generates a schedule that takes time and budget into consideration. When a user selects images and information that interest them, the server generates and suggests a travel plan based on that information.

[1172] The travel plan is displayed on the user's device and can be customized by the user. Based on the information adjusted by the user, the server adjusts and automatically generates the travel plan again.

[1173] The finalized itinerary is sent to the partner travel agency, where users can easily purchase and book the itinerary. Specifically, they are redirected to the travel agency's booking page and can complete the payment online.

[1174] Hardware and software used

[1175] Hardware: High-performance servers (e.g., Amazon EC2)

[1176] software:

[1177] API client: requests (for accessing APIs of social media platforms)

[1178] Image Recognition: Transformers (using CLIP model)

[1179] Generative AI: openai (using GPT-3)

[1180] Specific examples

[1181] For example, a user searches for information about "Tokyo travel" and selects a photo that catches their eye. Let's say the photo shows Tokyo Tower. Based on this information, the server generates a schedule that takes into account surrounding tourist spots, restaurants, and transportation options, including Tokyo Tower. If the user wishes to customize the schedule, the server can also accommodate requests such as adding Ueno Zoo and Shibuya.

[1182] Additionally, example prompts have the following format:

[1183] "Generate a travel plan based on the user's interests. Location: Tokyo Tower, Attractions: Ueno Zoo, Shibuya, Restaurant: High-end sushi restaurant, Activities: Sightseeing, Shopping"

[1184] In this way, the server is a system that automatically generates highly accurate travel plans based on the information obtained and provides them to users.

[1185] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1186] Step 1:

[1187] The server automatically collects information from social media sites based on specific keywords and hashtags. This process uses the social media site's API to retrieve images, videos, text, and metadata (location, number of likes, number of comments, etc.). The input is the keywords and hashtags specified by the user, and the output is the collected social media data.

[1188] Step 2:

[1189] The server extracts image data from the collected SNS data and applies image recognition technology to identify tourist attractions and stores. At this stage, the CLIP model is used to analyze the images. The input is image data obtained from SNS, and the output is information on the identified tourist attractions and stores.

[1190] Step 3:

[1191] Based on the identified information, the server uses generative artificial intelligence technology (GPT-3) to extract useful text data from post captions and comments. The input is the image data and text data of tourist attractions and stores identified in Step 2, and the output is the text data generated by the generative AI.

[1192] Step 4:

[1193] The server statistically analyzes the extracted data and classifies it into categories such as tourist attractions, restaurants, and events. The input is text data and metadata, and the output is information classified by category. The analysis uses major statistical methods.

[1194] Step 5:

[1195] The server designs a travel plan based on the classified information and automatically generates a schedule that takes time and budget into consideration. The input is information categorized by category, and the output is a detailed travel schedule. Specifically, it calculates time allocation and expenses by taking into account multiple destinations and activities.

[1196] Step 6:

[1197] The user can view the generated travel plan on their device and select images and information that interest them. The server then generates the optimal travel plan and presents it to the user. The input is the images and information selected by the user, and the output is a customized travel plan.

[1198] Step 7:

[1199] The user can customize the proposed itinerary through the terminal. The server reflects the user's changes and automatically regenerates the adjusted itinerary. The input is the user's customized information, and the output is the adjusted itinerary.

[1200] Step 8:

[1201] The server then sends the finalized itinerary to the affiliated travel agency. The user then completes the procedure to purchase and reserve the itinerary via the terminal. The input is the finalized itinerary, and the output is the data to be sent to the travel agency and a notification of reservation completion.

[1202] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1203] This invention relates to a system that automatically collects information from online social networking services (SNS) and analyzes it using image recognition and generative artificial intelligence technologies to provide users with optimal travel plans. Furthermore, by combining this with an emotion engine that recognizes the user's emotions, it provides users with a more personalized travel experience.

[1204] System configuration

[1205] This system mainly consists of a server, a user's device, and an emotion engine. The server collects information from SNS and analyzes and classifies the data using image recognition and generative artificial intelligence technology. The user's device receives the information, displays the travel plan, and allows for adjustments. The emotion engine also identifies the user's emotions and sends the data to the server.

[1206] Collection and analysis of SNS information

[1207] The server automatically collects posts based on specific keywords and hashtags using the APIs of social media platforms, including images, videos, posting dates and times, poster information, location information, hashtags, number of likes, and number of comments.

[1208] The server applies image recognition technology to the collected images and videos to identify place names, tourist attractions, stores, etc. It also uses generative artificial intelligence technology to extract relevant text data from post captions and comment sections, resulting in information categorized into tourist attractions, restaurants, events, etc.

[1209] Emotion Engine Functions

[1210] The user's device is equipped with a camera and microphone to recognize the user's emotions in real time. The emotion engine analyzes these sensor data and identifies the user's emotions (e.g., joy, surprise, excitement) from facial expressions and tone of voice. The emotion data recognized by the emotion engine is sent to a server for analysis.

[1211] Travel plan design and coordination

[1212] When a user clicks on an image or piece of information that interests them, the device sends that information to the server. Based on this, the server collects information on related tourist spots and stores, and automatically generates a travel plan that takes into account the user's preferences and emotional data. The generated plan also includes transportation and places to stay, and a schedule that takes time and budget into consideration is automatically created.

[1213] For example, if a user clicks on an image of a "trip to Paris" and expresses joy, the server will use that information to suggest a plan that combines popular tourist spots such as the Eiffel Tower and the Louvre.

[1214] View and rearrange your plan

[1215] The device displays the generated itinerary to the user, who can review it and make adjustments as needed, such as changing the order of the stops or selecting additional tourist attractions.

[1216] After the user adjusts the plan, the device sends the information back to the server, which then regenerates the plan based on the user's adjustments and the latest emotion data, and sends the final plan to the device.

[1217] Travel agency collaboration and booking

[1218] The server sends the finalized travel plan to the partner travel agency's system via API, allowing the user to purchase and reserve the travel plan directly. The terminal is then redirected to the travel agency's reservation page, where the user can complete payment procedures, etc. This allows the user to enjoy the optimal travel plan without any hassle.

[1219] As described above, the present invention provides a system that allows users to easily automatically generate optimal travel plans by utilizing information from SNS and combining it with user emotional data.

[1220] The processing flow will be explained below.

[1221] Step 1:

[1222] The server uses the API of a social networking service (SNS) to search for posts based on specific keywords or hashtags (e.g., "travel," "tourist destinations") The data collected through the search includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc.

[1223] Step 2:

[1224] The server stores the collected social media post data in a temporary database, allowing for efficient subsequent analysis. It also stores the metadata of each post for analysis.

[1225] Step 3:

[1226] The server analyzes the collected images using image recognition technology to identify key features (e.g., landmarks, tourist attractions, restaurants), leveraging computer vision techniques to recognize specific landmarks and objects.

[1227] Step 4:

[1228] The server uses generative artificial intelligence techniques to extract relevant text data (e.g., place names, store names, activity details) from post captions and comment sections, and analyzes the extracted text data using natural language processing (NLP) techniques to identify important keywords and phrases.

[1229] Step 5:

[1230] The server statistically analyzes the data extracted through image recognition and text analysis. Specifically, it calculates how frequently each feature appears and evaluates its popularity and relevance. Based on this, the data is classified into categories such as tourist attractions, restaurants, and events.

[1231] Step 6:

[1232] The server creates weekly, monthly, and yearly rankings based on statistically analyzed data. This allows users to easily understand popular spots and activities for each season. This ranking information is also used as reference when creating plans.

[1233] Step 7:

[1234] The camera and microphone on the user's device capture the user's facial expressions and tone of voice, which are then analyzed in real time by the emotion engine.

[1235] Step 8:

[1236] The emotion engine analyzes the user's emotion data (e.g., joy, surprise, excitement) and sends it to the server, where it is linked to the images and information the user is viewing.

[1237] Step 9:

[1238] When a user clicks on an SNS image or information that interests them, the device sends that information and linked emotional data to the server. Based on this, the server collects information on related tourist spots and stores, and automatically generates a travel plan taking into account the user's preference information and emotional data.

[1239] Step 10:

[1240] The server then designs a travel plan based on the user's emotional data, including activities that emphasize fun, based on a strong sense of "joy." The plan also includes transportation and accommodations, and automatically generates a schedule that takes into account time and budget.

[1241] Step 11:

[1242] The device displays the generated itinerary to the user, who can review it and make adjustments as needed, such as changing the order of the stops or selecting additional tourist attractions.

[1243] Step 12:

[1244] After the user adjusts the plan, the device sends the information back to the server, which then regenerates the plan based on the user's adjustments and the latest emotion data, and sends the final plan to the device.

[1245] Step 13:

[1246] The server sends the confirmed travel plan to the partner travel agency's system via API, allowing the user to purchase and reserve the travel plan directly. The terminal is then redirected to the travel agency's reservation page, where the user can complete payment procedures.

[1247] Example 2

[1248] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1249] Conventional travel plan creation systems have difficulty automatically generating personalized plans that reflect the user's emotions and individual preferences. Furthermore, there is a lack of a method for analyzing user emotions in real time and reflecting them in travel plans, making it difficult to improve the quality of the user experience.

[1250] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1251] In this invention, the server includes means for automatically collecting information from social media services on the Internet, means for identifying tourist attractions and stores from the collected information using image recognition technology, means for extracting the identified information as text data using generative artificial intelligence technology, means for collecting and analyzing emotional data in real time using an engine that recognizes user emotions, means for statistically analyzing the extracted data and the collected emotional data and classifying them by category, means for designing a travel plan based on the classified information and analyzed emotional data and automatically generating a schedule that takes time and budget into consideration, and means for displaying the generated travel plan on the user's device so that the user can check and adjust it. This enables the automatic generation of a personalized travel plan that takes into consideration the user's individual preferences and emotions.

[1252] "Internet social media service" refers to an online platform that enables users to share information and interact with others via the Internet.

[1253] "Image recognition technology" refers to the process of automatically identifying and identifying objects and text within an image using computer vision technology.

[1254] "Generative AI technology" refers to AI technology that has the ability to learn patterns from large amounts of data and generate new data and information.

[1255] "User emotion recognition engine" refers to technology that identifies emotions by analyzing a user's facial expressions and tone of voice.

[1256] "Statistical analysis" refers to the process of quantifying large amounts of data and analyzing its patterns and correlations using statistical methods.

[1257] "Categorizing" refers to the process of grouping collected data based on specific attributes or themes.

[1258] "Designing a travel plan" refers to the process of creating a travel schedule and itinerary by combining tourist attractions, activities, means of transportation, etc.

[1259] "Automatically generating a schedule that takes time and budget into consideration" refers to the process of automatically creating an optimal travel schedule using a computer based on the user's desired time period and budget restrictions.

[1260] "Displaying it on the user's device and allowing the user to check and adjust it" means displaying the generated travel plan on a device such as a smartphone or tablet, allowing the user to check and change its contents.

[1261] The present invention is a system that automatically collects information from social media services on the Internet and analyzes that information using image recognition technology and generative artificial intelligence technology to provide users with optimal travel plans. Furthermore, by combining it with an emotion engine that recognizes the user's emotions, the system provides a more personalized travel experience. Specific embodiments of the system are described below.

[1262] System configuration

[1263] This system consists of a server, a user's terminal, and an emotion engine.

[1264] Collecting SNS information

[1265] The server automatically collects posts based on specific keywords or hashtags using the APIs of social media platforms, such as Twitter API or Instagram Graph API. The collected data includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, and number of comments.

[1266] Image Recognition and Text Analytics

[1267] The server applies image recognition technology (e.g., Google Cloud Vision API) to the collected images and videos to identify place names, tourist attractions, stores, etc. It then uses generative artificial intelligence technology (e.g., OpenAI's GPT-4) to analyze posted captions and comments and extract related text data. This allows it to obtain information categorized into tourist attractions, restaurants, events, etc.

[1268] Emotion Engine Functions

[1269] The user's device is equipped with a camera and microphone, which the emotion engine uses to identify emotions in real time from the user's facial expressions and tone of voice. The emotion engine uses, for example, Microsoft Azure's Emotion API. The collected emotion data is sent to a server and used to plan the itinerary.

[1270] Travel plan design

[1271] When a user clicks on an image or piece of information that interests them, the server uses that information to collect information on related tourist spots and stores. Taking into account the user's emotional data and preference information, the server automatically generates a travel plan. The generated plan includes a schedule that takes into account transportation, places to stay, time, and budget. For example, if a user clicks on an image of a "trip to Paris" and expresses joy, the server will suggest a plan that includes popular tourist spots such as the Eiffel Tower, the Louvre, and the Champs-Élysées.

[1272] View and adjust your plan

[1273] The device displays the generated travel plan to the user. The user can review the plan, change the order of the destinations, or select additional tourist attractions. The adjusted plan information is sent back to the server, and the server regenerates the plan based on the user's adjustments and the latest emotion data, and sends the final plan to the device.

[1274] Book a travel plan

[1275] The server sends the finalized itinerary to the partner travel agency's system via API, and the terminal redirects the user to the travel agency's booking page. The user can then proceed with the procedure to purchase and reserve the itinerary, for example, using Expedia API or Booking.com API.

[1276] Examples and prompts

[1277] For example, "If a user clicks on an image of a trip to Paris and expresses joy, the server will suggest a travel itinerary that includes attractions such as the Eiffel Tower, the Louvre, and the Champs-Élysées."

[1278] An example of a prompt for a generative AI model is, "What is the best plan for a trip to Paris? Please list it in categories of tourist attractions, restaurants, and events."

[1279] The above is a detailed description of the mode for carrying out the invention.

[1280] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1281] Step 1: Collect social media information

[1282] Input: Specific keywords or hashtags

[1283] Specific operation: The server uses the API of the social media platform (e.g., Twitter API, Instagram Graph API) to collect posts that match the specified keywords or hashtags.

[1284] Output: Images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc.

[1285] Step 2: Image Recognition

[1286] Input: Collected image and video data

[1287] Specific operation: The server uses image recognition technology (e.g., Google Cloud Vision API) to identify place names, tourist attractions, stores, etc. from this data.

[1288] Output: Information on identified tourist attractions and stores, and corresponding image and video data

[1289] Step 3: Text analysis

[1290] Input: Collected post captions and comments

[1291] How it works: The server uses generative artificial intelligence techniques (e.g., OpenAI's GPT-4) to extract relevant text data from captions and comments and classify them into categories such as tourist attractions, restaurants, and events.

[1292] Output: Extracted text data and categorical information

[1293] Step 4: Collecting emotion data

[1294] Input: User camera and microphone data

[1295] How it works: The device uses an emotion engine (e.g., Microsoft Azure Emotion API) to identify emotions in real time from the user's facial expressions and tone of voice.

[1296] Output: Collected emotion data, identified emotion types

[1297] Step 5: Analyze the sentiment data

[1298] Input: Collected emotion data

[1299] Specific operation: The server analyzes the collected emotional data to identify the user's current emotional state. Based on this analysis result, it is used to generate a travel plan.

[1300] Output: Parsed emotion data

[1301] Step 6: Generate your travel plan

[1302] Input: Identified tourist destination information, extracted text data, analyzed emotion data, user preference information

[1303] Specific operation: Based on this input data, the server collects information on tourist spots and stores, and automatically generates a travel plan taking into account the user's emotional data and preference information.

[1304] Output: Automatically generated travel plan, schedule information, transportation, and accommodation

[1305] Step 7: View your travel plans

[1306] Input: Auto-generated itinerary

[1307] Specific operation: The terminal displays the generated travel plan to the user so that the user can confirm it.

[1308] Output: Travel plan displayed on the user's device

[1309] Step 8: Adjust your travel plans

[1310] Input: User adjustment instructions

[1311] Specific operation: The user checks the displayed plan, changes the order of the stops, or selects additional tourist attractions. The adjustments are sent to the server via the device.

[1312] Output: Adjusted travel plan instructions

[1313] Step 9: Regenerate the adjusted plan

[1314] Input: Adjusted travel plan instructions, latest sentiment data

[1315] Specific operation: The server generates a new travel plan based on these input data and sends the final plan to the terminal.

[1316] Output: Final itinerary

[1317] Step 10: Book your travel plan

[1318] Input: Final itinerary

[1319] Specific operation: The server sends the finalized travel plan to the partner travel agency's system via API, and the terminal redirects the user to the travel agency's booking page, where the user completes the booking procedure.

[1320] Output: Confirmed reservation information, redirect to reservation completion page

[1321] The above is a detailed description of the specific steps of the program processing of this system.

[1322] (Application example 2)

[1323] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1324] Conventional travel plan providing systems have the problem of being unable to flexibly respond to users' emotions and individual needs. They also lack a means to effectively communicate travel plans to users in an easy-to-understand manner. As a result, users end up spending a lot of time and effort selecting travel destinations and planning their schedules.

[1325] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for automatically collecting information from social networking services on the Internet, means for identifying tourist spots and stores from the collected information using image recognition technology, and means for extracting the identified information as text data using generative artificial intelligence technology. This makes it possible to recognize the user's emotions and personalize the travel plan using the emotion data. Furthermore, by adding means for generating a travel plan based on emotions in video format, the travel plan can be communicated to the user visually and effectively.

[1326] An "online social networking service" is a platform that allows users to share information and communicate with each other via the Internet.

[1327] "Means of automatically collecting information" refers to a method of automatically obtaining posted data from social networking services on the Internet using specific keywords or hashtags through a program.

[1328] "Image recognition technology" is a technology in which a computer analyzes image data and identifies specific objects or landmarks.

[1329] "Generative artificial intelligence technology" is an artificial intelligence technology that analyzes collected data and automatically generates text and data.

[1330] "Means of extracting as text data" refers to a method of extracting relevant text information from collected data using image recognition technology or generative artificial intelligence technology and providing it in a meaningful format.

[1331] "Means of statistical analysis and categorization" refers to a method of analyzing extracted data using statistical analysis and dividing it into categories such as tourist attractions, stores, and events based on their relevance.

[1332] "A means for designing travel plans and automatically generating schedules that take time and budget into consideration" is a method for creating travel plans based on classified information and automatically generating the optimal schedule to meet the user's requirements.

[1333] "Means for displaying on the user's device and allowing the user to review and adjust" refers to a method for displaying the generated travel plan on the user's smartphone or computer, allowing the user to review the plan and make changes as necessary.

[1334] "Means for recognizing a user's emotions and using that emotional data to personalize travel plans" refers to a method for identifying a user's emotions by analyzing their facial expressions and tone of voice, and then using that data to provide travel plans that match the user's preferences.

[1335] The "means for generating emotion-based travel plans in video format" is a method for generating videos and providing them to users based on information about tourist spots and events that take into account the user's emotional data.

[1336] The present invention relates to a system that collects information from social networking services (SNS) and provides travel plans in the form of videos that take into account the user's emotions. Specific embodiments of the system will be described below.

[1337] This system mainly consists of a server, a user's device, and an emotion engine. The server includes a social networking data collection module, a data analysis module, a travel plan generation module, and a video generation module. The user's device is equipped with a camera and microphone and includes an emotion engine for emotion recognition.

[1338] Program Overview

[1339] 1. Collecting social media data

[1340] The server uses APIs from social media platforms to automatically collect data based on specific keywords and hashtags, including images, videos, posting dates and times, poster information, location information, hashtags, number of likes, number of comments, etc. Specific software used includes social media APIs and data collection libraries.

[1341] 2. Data Analysis

[1342] The collected data is then analyzed using image recognition technology (e.g., OpenCV and TensorFlow) to identify tourist attractions and stores. Generative AI technology is then used to extract relevant text data from post captions and comment sections, allowing each tourist attraction and store to be classified into a category.

[1343] 3. Emotional Recognition

[1344] The user's device is equipped with a camera and microphone, and by analyzing these sensor data, the emotion engine recognizes the user's emotions in real time. The emotion engine identifies whether the user is expressing emotions such as joy or surprise from facial expressions and tone of voice. This data is sent to a server and used to customize the travel plan. Specific software includes emotion recognition libraries (e.g., FaceReader and Vokaturi).

[1345] 4. Designing a travel plan

[1346] The server designs a travel plan based on the information the user clicks on and their emotional data. The plan includes sightseeing spots, shops, events, etc., and the schedule is automatically generated taking into account time and budget. The generated plan can be adjusted to suit the user's preferences.

[1347] 5. Video Generation and Display

[1348] The finalized itinerary is then generated in video format by a video generation module. The generated video is sent to the user's device, where it presents the itinerary in a visually easy-to-understand format. Specific software includes a video editing library (e.g., FFmpeg, MoviePy, etc.).

[1349] Specific examples

[1350] For example, if a user expresses interest in a trip to Paris, the system collects data related to Paris from social media and analyzes information on tourist attractions and restaurants. If the user clicks on an image of Paris on the screen and expresses joy, the server will suggest a travel plan that combines popular spots such as the Eiffel Tower, the Louvre, and Montmartre. A video based on the plan is generated and provided to the user.

[1351] Prompt Sentence Examples

[1352] "If the user's analyzed emotion is detected as 'surprise,' generate a video of 'recommended travel destinations full of surprises.' For example, suggest a plan that includes unique experiences such as an underwater hotel or a trapeze."

[1353] As described above, the system of the present invention can provide users with optimal travel plans in video format by combining information collected from SNS with user emotion data.

[1354] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1355] Step 1:

[1356] The server automatically collects data from social networking services on the Internet using keywords and hashtags. Keywords that reflect the user's interests (e.g., "trip to Paris") are used for this input. The collected data includes images, videos, posting date and time, poster information, location information, hashtags, number of likes, number of comments, etc. The specific software used to collect the data is the SNS API or data collection library. The server obtains this posted data.

[1357] Step 2:

[1358] The server uses image recognition technology (such as OpenCV or TensorFlow) on the collected data to identify tourist attractions and stores. The input is the image data collected in step 1, and the output is text data in which tourist attractions and stores are identified. Image recognition technology is used to identify tourist attraction landmarks, store logos, etc., and the identified information is extracted as text.

[1359] Step 3:

[1360] The server uses generative AI technology to extract relevant text data from post captions and comment sections. The input is the text data of the social media post, and the output is the analyzed text data. Generative AI technology is used to understand the content of the post and extract information about tourist spots, stores, and events. Specifically, a natural language processing library is used.

[1361] Step 4:

[1362] The server statistically analyzes the extracted text data and classifies it into categories such as tourist attractions, stores, and events. The input is the text data extracted in step 3, and the output is information classified by category. The server analyzes the data using a statistical analysis library and classifies it into each category.

[1363] Step 5:

[1364] The server uses the user's emotional data to design a travel plan and automatically generate a schedule that takes time and budget into consideration. The input is the information classified in step 4 and the user's emotional data. The emotion engine uses the user's camera and microphone to obtain emotional data in real time (e.g., joy, surprise, etc.) and generates a travel plan based on those emotions. Specifically, the emotion engine uses FaceReader and Vokaturi.

[1365] Step 6:

[1366] The server generates a travel plan based on emotions in video format and sends it to the user's device. The input is the travel plan generated in step 5, and the output is a video travel plan. Using a video generation module (e.g., FFmpeg or MoviePy), videos of tourist spots and events according to emotions are created and visually presented to the user.

[1367] Step 7:

[1368] The device displays the generated video to the user, who then reviews the plan and adjusts it as necessary. The input is the video generated in step 6, and the output is the travel plan adjusted by the user. The user can review the travel plan through the device and make adjustments, such as changing the order of the destinations. The adjusted information is sent back to the server, and the plan is regenerated as necessary.

[1369] The above is a description of the specific processing steps of the system that realizes the application example.

[1370] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1371] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1372] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1373] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1374] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1375] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1376] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1377] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1378] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1379] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1380] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1381] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1382] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1383] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1384] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1385] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1386] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1387] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1388] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1389] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1390] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1391] The following is further disclosed regarding the above embodiment.

[1392] (Claim 1)

[1393] A means for automatically collecting information from social networking services on the Internet;

[1394] A means of identifying tourist spots and stores from the collected information using image recognition technology,

[1395] A means for extracting the identified information as text data using generative artificial intelligence technology;

[1396] A means of statistically analyzing the extracted data and classifying them into categories;

[1397] A means to design a travel plan based on the classified information and automatically generate a schedule that takes time and budget into consideration;

[1398] means for displaying the generated itinerary on the user's device to allow the user to review and adjust it;

[1399] A system including:

[1400] (Claim 2)

[1401] 2. The system according to claim 1, further comprising means for automatically re-creating an adjusted travel plan based on adjustment information selected by the user.

[1402] (Claim 3)

[1403] A means for transmitting the finalized travel plan to a partner travel agency;

[1404] 10. The system of claim 1, further comprising means for providing a process for a user to purchase and reserve a travel plan.

[1405] "Example 1"

[1406] (Claim 1)

[1407] A means for automatically collecting information from social networking services on the Internet;

[1408] A means of identifying tourist spots and stores from the collected information using image recognition technology,

[1409] A means for extracting the identified information as text data using generative artificial intelligence technology;

[1410] A means of statistically analyzing the extracted data and classifying them into categories;

[1411] A means to design a travel plan based on the classified information and automatically generate a schedule that takes time and budget into consideration;

[1412] means for displaying the generated itinerary on the user's device to allow the user to review and adjust it;

[1413] A means for automatically generating new travel plans based on user-selected information;

[1414] means for transmitting the confirmed travel itinerary to affiliated travel providers and providing a process for users to book and purchase the travel itinerary;

[1415] A system including:

[1416] (Claim 2)

[1417] 2. The system according to claim 1, further comprising means for automatically re-creating an adjusted travel plan based on adjustment information selected by the user.

[1418] (Claim 3)

[1419] 10. The system of claim 1, further comprising means for transmitting the finalized itinerary to affiliated travel providers and providing a process for the user to purchase and reserve the itinerary.

[1420] "Application Example 1"

[1421] (Claim 1)

[1422] A means for automatically collecting information from social networking services on the Internet;

[1423] A means of identifying tourist spots and stores from the collected information using image recognition technology,

[1424] A means for extracting the identified information as text data using generative artificial intelligence technology;

[1425] A means of statistically analyzing the extracted data and classifying them into categories;

[1426] A means to design a travel plan based on the classified information and automatically generate a schedule that takes time and budget into consideration;

[1427] A means for users to generate optimal travel plans by selecting images and information that interest them;

[1428] A way for users to customize the travel plans suggested by the system, and

[1429] A means to transmit final itineraries to partner travel providers and facilitate booking and payment procedures;

[1430] A system including:

[1431] (Claim 2)

[1432] 10. The system of claim 1, further comprising means for automatically generating a travel plan after adjusting the travel plan.

[1433] (Claim 3)

[1434] 10. The system of claim 1, further comprising means for transmitting the final itinerary to a partner travel provider and providing a process for the user to purchase and reserve the itinerary.

[1435] "Example 2: Combining Emotion Engines"

[1436] (Claim 1)

[1437] means of automatically collecting information from social media services on the Internet;

[1438] A means of identifying tourist spots and stores from the collected information using image recognition technology,

[1439] A means for extracting the identified information as text data using generative artificial intelligence technology;

[1440] A means for collecting and analyzing emotion data in real time using an engine that recognizes user emotions;

[1441] a means for statistically analyzing the extracted data and collected emotion data and classifying them into categories;

[1442] A means for designing a travel plan based on the classified information and analyzed emotion data, and automatically generating a schedule that takes time and budget into consideration;

[1443] means for displaying the generated itinerary on the user's device to allow the user to review and adjust it;

[1444] A system including:

[1445] (Claim 2)

[1446] 2. The system according to claim 1, further comprising means for automatically re-creating an itinerary by adjusting it based on adjustment information selected by the user.

[1447] (Claim 3)

[1448] A means for transmitting the finalized travel plan to a partner travel agency;

[1449] 10. The system of claim 1, further comprising means for providing a process for a user to purchase and reserve a travel plan.

[1450] "Application example 2 when combining emotion engines"

[1451] (Claim 1)

[1452] A means for automatically collecting information from social networking services on the Internet;

[1453] A means of identifying tourist spots and stores from the collected information using image recognition technology,

[1454] A means for extracting the identified information as text data using generative artificial intelligence technology;

[1455] A means of statistically analyzing the extracted data and classifying them into categories;

[1456] A means to design a travel plan based on the classified information and automatically generate a schedule that takes time and budget into consideration;

[1457] means for displaying the generated itinerary on the user's device to allow the user to review and adjust it;

[1458] A means of recognizing user emotions and using that emotional data to personalize travel plans;

[1459] A means for generating emotion-based travel plans in video format;

[1460] A system including:

[1461] (Claim 2)

[1462] 2. The system according to claim 1, further comprising means for automatically re-creating an adjusted travel plan based on adjustment information selected by the user.

[1463] (Claim 3)

[1464] A means for transmitting the finalized travel plan to a partner travel agency;

[1465] 10. The system of claim 1, further comprising means for providing a process for a user to purchase and reserve a travel plan. [Explanation of symbols]

[1466] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for automatically collecting information from social networking services on the Internet; A means of identifying tourist spots and stores from the collected information using image recognition technology, A means for extracting the identified information as text data using generative artificial intelligence technology; A means of statistically analyzing the extracted data and classifying them into categories; A means to design a travel plan based on the classified information and automatically generate a schedule that takes time and budget into consideration; means for displaying the generated itinerary on the user's device to allow the user to review and adjust it; A system including:

2. 2. The system according to claim 1, further comprising means for automatically generating a new travel plan by adjusting it based on adjustment information selected by the user.

3. A means for transmitting the finalized travel plan to a partner travel agency; 10. The system of claim 1, further comprising means for providing a process for a user to purchase and reserve a travel plan.

Images