system

Abstract

We provide the system. [Solution] Means of obtaining user interest information, Means for obtaining the user's location information, A means for generating music and guidance text based on the aforementioned interest information and location information, A means of generating a musical experience by combining the aforementioned music and explanatory text, The means of providing the aforementioned musical experience, A system that includes this.

Description

Technical Field

[0001] The technology of this disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, lack of exercise is recognized as a problem that causes serious health risks. Many people have insufficient exercise in their daily lives and need to actively move their bodies to maintain health. However, it is difficult to maintain motivation with conventional exercise methods, and there is a problem that many people cannot continue to exercise. The purpose of this invention is to provide a new approach that strengthens the motivation to go out and eliminates lack of exercise while having fun.

Means for Solving the Problems

[0005] This invention provides means for acquiring user interest information and location information, and means for generating music and guidance text based on that information. By combining the generated music and guidance text to create a musical experience and providing it to the user, a system is constructed that allows users to enjoy themselves both visually and aurally while walking and exploring the city. In addition, the system provides users with information on nearby spots as navigation and offers new experiences through real-time information updates. Furthermore, by collecting user feedback and improving the generation algorithm, it is possible to continuously provide content that matches the user's interests and increase their motivation to exercise.

[0006] "User interest information" refers to data about the fields and themes that individual users are interested in, and is collected from past search history and profile information.

[0007] "Location information" refers to data that indicates the user's current geographical location, and is obtained through GPS or other positioning methods.

[0008] "Means for generating music and explanatory text" refers to technical methods and devices for creating music and explanatory text related to user interests and destinations, based on data obtained from users.

[0009] A "musical experience" refers to an event or activity that combines generated music and explanatory text to provide users with a visually and aurally enjoyable experience.

[0010] "Nearby Spot Information" refers to information about interesting places and facilities located around the user's current location.

[0011] "Means of providing navigation" refers to methods and devices for efficiently presenting users with routes and related information to their destination.

[0012] "Means of collecting user feedback" refers to a system for obtaining user impressions and evaluations based on their experiences and storing them as data.

[0013] "Means of improving generation algorithms" refers to technologies that improve and optimize the content generation process based on collected data. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of the data processing device and smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when the emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when the emotion engine is combined.

Mode for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the language used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be one arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be one type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include CPU (Central Processing Unit), GPU (Graphics Processing Unit), GPGPU (General-Purpose computing on Graphics Processing Units), APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disk (e.g., hard disk), or magnetic tape, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] Embodiments of the present invention relate to a system that utilizes user interest information and location information to provide a personalized musical experience. The system mainly consists of a server, a terminal, and a user.

[0036] The server receives location and interest information transmitted from the user's smartphone or other device, and analyzes this data using a generative AI model. Based on the analyzed data, it is responsible for generating music content and navigation instructions that match the user's interests at that time. For example, if the user enjoys visiting cafes, it will create an original song about popular cafes in the area, along with corresponding directions.

[0037] The terminal plays musical content received from the server through the user interface. The combination of music and explanatory text allows users to enjoy learning while exploring the city. For example, a user visiting Shinjuku might receive a musical-style guide to Cafe Street through the terminal, followed by navigation to their next destination.

[0038] Users experience content provided through an application on their device. Following the provided music and guidance, users can increase their motivation to visit new places while also addressing a sedentary lifestyle. During this experience, users receive real-time information and contribute to improving future experiences by providing feedback.

[0039] As described above, the system of the present invention utilizes the user's location and individual interest data to provide a personalized and highly interactive experience, aiming to promote the alleviation of sedentary lifestyles in a gamified manner.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The device uses GPS to obtain the user's current location when the application is launched, and extracts interest information from past search history and user profile. This data is encrypted and sent to the server.

[0043] Step 2:

[0044] The server receives location and interest information transmitted from the terminal. Using this information as input data, it starts content generation processing based on an AI model.

[0045] Step 3:

[0046] The server aggregates relevant geographical point and spot information based on the user's current location. Simultaneously, it generates music content and guide text tailored to the user's interests.

[0047] Step 4:

[0048] The server integrates the generated music and explanatory text to create a musical experience content for the user, and then sends that data to the terminal.

[0049] Step 5:

[0050] The device plays musical experience content received from the server. This allows the user to explore the city through sight and sound.

[0051] Step 6:

[0052] Users navigate the city following the content provided via their devices. They enjoy navigation to their destination accompanied by music through a musical experience.

[0053] Step 7:

[0054] The device collects user feedback and sends it to the server. The server analyzes this feedback and uses it to improve the generation algorithm, which is then reflected in future content generation.

[0055] (Example 1)

[0056] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0057] In modern society, there is a demand for information tailored to individual users. However, conventional music streaming and navigation systems have struggled to provide personalized experiences that match individual users' interests and location information. Furthermore, they lacked mechanisms to reflect user feedback in real time and continuously improve the experience. As a result, there is a problem of decreased user experience value and inability to increase satisfaction.

[0058] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0059] In this invention, the server includes means for acquiring user interest information, means for acquiring user location information, and means for generating prompt data using generative artificial intelligence based on the interest information and location information, and for generating music and guidance text. This makes it possible to provide each user with a personalized musical experience based on their interests and current location information.

[0060] A "user" refers to an individual who uses a system and experiences it.

[0061] "Interest information" refers to data about specific fields or activities that a user is interested in.

[0062] "Location information" refers to geographical data that indicates the user's current location.

[0063] "Generative artificial intelligence" refers to a computer system that generates information based on large datasets, using natural language processing and other techniques.

[0064] "Prompt data" refers to the input data used by generative artificial intelligence to generate information.

[0065] "Music" refers to auditory content generated for the user.

[0066] "Information text" refers to text data intended to provide navigation and information to users.

[0067] A "musical experience" refers to an interactive experience that combines music and explanatory text.

[0068] "User interface" refers to the screens and other means of interaction that users use when directly interacting with a system.

[0069] "Feedback" refers to information that users provide to the system to evaluate their experience and offer their opinions.

[0070] This system is designed to provide users with a more personalized musical experience. It primarily consists of servers, terminals, and users.

[0071] The server's role is to analyze user interest and location information using a generative AI model to generate personalized music and navigation text. The server is located in a cloud environment, and the generative AI model used is one that excels at natural language generation. For example, a large-scale language model such as GPT-4® could be considered. This generative AI model takes a prompt sentence as input and generates appropriate music content and navigation guidance. For example, a prompt sentence might be in the format of "The user is currently in Shinjuku and likes cafes. Please generate musical songs about nearby cafes."

[0072] The terminal plays the role of playing content sent from the server. Terminals include smartphones and tablet devices, and users receive music and navigation guidance through applications. By playing music on the user interface while displaying guidance text on the screen, it makes it easier for users to explore the city.

[0073] Users visit new locations by following music and instructions provided through their devices. This format encourages users to explore in an enjoyable and meaningful way, and allows for real-time feedback. User feedback is collected by the server to improve future experiences and helps refine the algorithms of the generative AI model.

[0074] This invention provides users with a more personalized and interest-based interactive experience. This allows us to meet the modern need for personalized information while simultaneously improving user satisfaction.

[0075] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0076] Step 1:

[0077] The server receives interest information and location information from the user's device. Based on this input data, it understands what the user is interested in and their current geographical location. Specifically, it utilizes GPS data provided by the device and interest information specified within the app, and aggregates this information on the server. The server's output is the interest information and location information for each user, which is necessary for analysis.

[0078] Step 2:

[0079] The server generates a prompt based on the received interest and location information. This prompt is then formatted as data to be input to the generative AI model. Specifically, it creates a sentence in a format easily understood by the generative AI model, such as, "The user is currently in △△ and likes □□. Please generate musical songs about the surrounding □□." The server's output is this prompt.

[0080] Step 3:

[0081] The server inputs the generated prompt text into a generation AI model to generate music and navigation text. Based on the prompt text provided as input, the generation AI model uses its built-in algorithms to generate music content and navigation guidance that corresponds to the user's interests. For example, if the user is assumed to like cafes, music and directions related to nearby cafes will be generated. The server's output consists of the generated music data and navigation text.

[0082] Step 4:

[0083] The server sends the generated music content and navigation instructions to the terminal. During this process, the data is encrypted to ensure secure transmission. The terminal then appropriately analyzes the data received from the server and prepares it for display as output on the user interface.

[0084] Step 5:

[0085] The device plays received music content and displays navigation instructions. Specifically, while the user explores the city, the device plays music through earphones or speakers, and displays the next destination and route information on the screen. The user can then act accordingly.

[0086] Step 6:

[0087] Users explore and experience the provided content, providing feedback on their devices. Users evaluate the quality of the experience and the accuracy of the guidance, and the device sends this data to the server. This feedback is used to improve the algorithm of the next generation AI model. The output is feedback data.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] In modern society, users are expected to efficiently acquire information that matches their preferences and enjoy fulfilling experiences. However, existing information provision systems struggle to provide flexible guidance tailored to individual interests and to deliver personalized, immersive experiences. In particular, in spaces visited by users with diverse interests, there is a need to provide new experiences that integrate music and navigation based on individual preferences.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes means for acquiring user preference information, means for acquiring user location data, and means for generating music and guidance information based on the preference information and location data. This makes it possible for users to enjoy a personalized musical experience in which they can enjoy music and guidance information tailored to their individual interests.

[0093] "User preference information" refers to information about the interests and concerns of individual users, specifically the distinctive preferences they show regarding particular music genres or places.

[0094] "Location data" refers to geographical information about the user's current location or the location they are planning to move to, and is obtained using GPS or other location measurement technologies.

[0095] "Means for generating music and navigation information" refers to a system that has the function of creating music content and navigation information optimized for the user based on collected user preference information and location data.

[0096] "Means of constructing a musical experience" refers to methods and processes for integrating generated music and informational data to create a new sensory experience for users.

[0097] "Means of delivering a musical experience" refers to the system for actually delivering a constructed musical experience to users, and is usually provided to users through terminals or devices.

[0098] "A spatial route designed based on user interests" refers to a spatial design that takes into account the user's preferences and location data to create a movement path optimized for that individual.

[0099] "Methods for generating custom music" refers to technologies that create music individually tailored to specific situations and interests, based on information obtained from the user's preferences and location.

[0100] In embodiments of the present invention, the entire system consists of the user's smart device and a server. The server acquires location data and preference information transmitted from the user's device and generates music and guidance information using a generative AI model based on this data. The generated content is transmitted to the user's device and provided there as a musical experience.

[0101] The hardware used will be the user's smartphone or mobile device, which will acquire location information. Additionally, the server running the generative AI model will operate on a cloud service. Specific software used will include Python for data analysis and OpenAI®, a machine learning library for generative AI models.

[0102] As a concrete example, when a user visits a large shopping center, information on stores that the user is interested in is compiled, and based on that information, a personalized musical guide is generated for the user. This guide combines music that matches the user's interests with store information, supporting a more enjoyable and efficient exploration.

[0103] An example of a prompt message is: "Generate an original musical guide based on the user's interests and design the most appropriate in-store route based on the specified coordinates. Interests: {interest information}, Current location: {location information}". By inputting this prompt message into the AI ​​model, the system provides the user with the most suitable content in real time.

[0104] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0105] Step 1:

[0106] The user's device obtains its current location. It uses GPS to collect the user's location data and sends it to the server. The input is location coordinate data, and the output is location information sent to the server. Specifically, the device's internal location service obtains the location in the background, updates it in real time, and continuously sends it to the server.

[0107] Step 2:

[0108] The server retrieves user preference information. The server obtains individual preference data from the user profile or selected interest categories. The input is user interest information, and the output is an interest dataset. The process involves individually evaluating user preferences based on data periodically sent from the device, and accessing a database on the server to retrieve interest information.

[0109] Step 3:

[0110] The server uses location data and preference information to generate music and navigation information using a generative AI model. The input is location data and preference information, and the output is personalized music content and navigation information. The generative AI model on the server analyzes this input data and generates appropriate content through prompts.

[0111] Step 4:

[0112] The server sends the generated content to the user's terminal. The server transfers data packets containing music and guidance information to the user's device. The output is the data packets received by the terminal. The data transmitted from the server is efficiently packetized according to the network protocol and delivered to the terminal.

[0113] Step 5:

[0114] The system plays music and navigation information received by the user's device as a musical experience. Input is data packets sent from the server, and output is music and navigation received by the user aurally and visually. The device plays the music using an audio player and displays navigation information on the screen.

[0115] Step 6:

[0116] Users enter feedback on the experience provided. This feedback is collected and sent to the server. The input is user feedback, and the output is feedback data used for future generation. Users enter their opinions into forms or questionnaires displayed on their devices, and upon completion, this information is registered in the database.

[0117] Step 7:

[0118] The server utilizes user feedback to improve its generation algorithm. The input is feedback data, and the output is the improved algorithm. By analyzing the feedback information collected on the server and updating the training dataset for the generation AI model, the user can further personalize their experience in subsequent uses.

[0119] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0120] Embodiments of the present invention relate to a system that provides a more personalized entertainment experience by offering a musical experience adapted to the user's emotional state. This system includes terminal, server, and user components, and further incorporates an emotion engine that performs emotion recognition.

[0121] The device acquires emotional data such as the user's facial expressions and voice tone. This data is analyzed in real time within the device or through a connected emotion engine. The results are sent to the server as the user's emotional state. This emotional information, along with other interest information and location information, is also sent to the server.

[0122] The server uses a generative AI model to generate music and guidance text tailored to the user, based on received emotional, interest, and location information. Furthermore, the emotional engine uses data analyzing the user's emotions to dynamically adjust the music's tone and tempo, as well as the content of the guidance text, creating content that best suits the user's current state. For example, if the user is relaxed, it provides calm music and soothing guidance text; if they are energetic, it generates energetic music and dynamic guidance text.

[0123] The server then sends the tailored musical experience content to the device. Through this content, played on the device, users can proactively enjoy a rich experience rooted in their emotions. Users can enjoy music synchronized with their emotions while using their smartphones for everyday commutes and sightseeing, and their interest is further piqued by real-time changing guided text.

[0124] Thus, the present invention opens a new path to making the elimination of sedentary lifestyle enjoyable, while maintaining emotional engagement tailored to the individual user's situation.

[0125] The following describes the processing flow.

[0126] Step 1:

[0127] The device, with the user's permission, uses the smartphone's camera and microphone to capture the user's facial expressions and voice, thereby acquiring emotional data. The emotion engine analyzes this data in real time to identify the user's emotional state.

[0128] Step 2:

[0129] The device transmits user interest information and location information to the server along with acquired emotional state data. This information serves as foundational data for building a personalized user experience.

[0130] Step 3:

[0131] The server comprehensively analyzes received emotional, interest, and location information. It then uses an AI model to generate music content and guidance text best suited to the user's current situation. During this process, the atmosphere of the music and the tone of the guidance text are dynamically adjusted according to the user's emotional state.

[0132] Step 4:

[0133] The server sends the generated, personalized musical experience content to the device. This content is optimized based on the user's past history and current emotions.

[0134] Step 5:

[0135] The device plays content received from the server, providing users with a musical experience. Users can explore the city while on the go through music and guidance synchronized with their emotions, and enjoy real-time updated information.

[0136] Step 6:

[0137] Users can provide feedback based on their experience through their device. The device sends this feedback to the server. The server analyzes the collected feedback and uses it to improve algorithms that further enhance the user experience in future content creation.

[0138] (Example 2)

[0139] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0140] Modern entertainment systems face challenges in providing individually customized content based on a user's emotional state, potentially leading to decreased user satisfaction. Furthermore, there is a lack of systems that effectively utilize user interest and location information to deliver optimal entertainment in real time. This invention aims to improve the personalized entertainment experience by considering the user's emotional state and generating more individualized voice and guidance text.

[0141] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0142] In this invention, the server includes means for acquiring user emotional data, means for analyzing the acquired emotional data, and means for acquiring user interest information. This makes it possible to personalize voice and guidance text based on the user's emotional state and provide a user-optimized entertainment experience in real time.

[0143] "Emotional data" refers to information that indicates a user's emotional state, obtained from things like their facial expressions and tone of voice.

[0144] "Analysis" refers to the process of processing acquired emotional data to identify or classify the user's emotions.

[0145] "Interest information" refers to information about a user's interests and preferences, and is data used to generate personalized entertainment experiences.

[0146] "Location information" is data that indicates the user's current geographical location and is used to provide content based on the user's movement and surrounding environment.

[0147] "Sound" refers to music and sounds generated based on the user's emotional state, and is an element that enriches the user's experience.

[0148] An "informative text" is a text or audio message associated with audio that provides information to the user.

[0149] An "entertainment experience" is a series of user experiences that combine customized audio and guidance text generated based on emotional data, interest information, and location information.

[0150] "Real-time" refers to a process where information processing and content generation are performed in response to the user's current state.

[0151] This invention is a system that provides personalized entertainment experiences based on the user's emotional state. The system consists of a terminal, a server, and a user, and utilizes the user's emotions, interests, and location information to generate optimal voice and guidance text.

[0152] The terminal is a device equipped with a camera and microphone as a means of acquiring user emotional data. Specifically, it can utilize common electronic devices such as smartphones and tablets. The terminal uses the sensors of these devices to collect the user's facial expressions and voice tone in real time and analyzes them using an emotion engine. This analysis makes it possible to understand the user's emotional state.

[0153] The server receives emotional information, interest information, and location information transmitted from the terminal, and combines this data to generate voice and guidance text using a generative AI model. The generative AI model can utilize, for example, open-source natural language processing models or commercial AI services. This makes it possible to adjust the voice tone and guidance text content according to the user's emotions.

[0154] Users utilize this system while carrying their devices and moving around. For example, by going out with their smartphone and walking around, they can receive music that matches their mood and real-time guidance. This allows users to enjoy a richer experience. An example of a specific prompt message is sending the instruction, "Generate calming music suitable for when the user is relaxed," to the generation AI model. Based on such prompt messages, the system can provide optimal content that meets the user's needs.

[0155] Thus, this system is designed to provide a dynamic and personalized entertainment experience that takes into account the user's emotions and activities.

[0156] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0157] Step 1:

[0158] The device uses a camera and microphone to acquire emotional data such as the user's facial expressions and voice tone. The input is the user's physical facial expressions and voice, while the output is digitized emotional data. This data is collected in real time for use in subsequent analysis steps.

[0159] Step 2:

[0160] The device analyzes the acquired emotional data using an emotion engine. This analysis uses a machine learning model to evaluate facial feature points and voice frequency components to identify the emotional state. The input is the emotional data obtained in step 1, and the output is an emotion label such as "happy," "sad," or "surprised." This quantifies the user's emotional state.

[0161] Step 3:

[0162] The device collects user interest information and location information along with emotional information, and sends them to the server. The input for this step is the analyzed emotional information, pre-configured interest information, and location information obtained via GPS, etc., and the output is the data to be sent to the server. The data is transmitted securely through a secure communication method.

[0163] Step 4:

[0164] The server uses the received emotion, interest, and location information to send prompt messages to the generating AI model. This input is the combined information obtained in step 3, and the model generates voice and guidance text. The output is an audio file and guidance text optimized for the user's situation. This process creates personalized content based on the user's emotions.

[0165] Step 5:

[0166] The server sends the generated audio and guidance text to the terminal. The input for this step is the generated content obtained in step 4, and the output is its transfer to the terminal. The terminal receives this transmitted data and begins playing it at a time appropriate for the user.

[0167] Step 6:

[0168] The terminal plays the transmitted audio and guidance text, providing the user with an entertainment experience. The input is the audio file and guidance text received in step 5, and the output is the audiovisual content that the user actually experiences. In this step, the playback content may be dynamically adjusted in response to the user's reactions and the environment while moving.

[0169] (Application Example 2)

[0170] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0171] Traditional entertainment experiences have struggled to provide content tailored to individual user emotions. As a result, users are often provided with experiences that are not necessarily optimal, failing to fully realize the appeal of entertainment. In particular, virtual environments often lack a sense of presence and immersion, making it essential to improve user engagement by providing personalized experiences.

[0172] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0173] In this invention, the server includes means for acquiring user interest information, means for acquiring user location information, means for analyzing user emotional information, and means for generating music and guidance text based on the interest information, location information, and emotional information. This enables a personalized musical experience synchronized with the user's emotions, enhancing realism and immersion, and providing a more individualized and optimized entertainment experience.

[0174] "User interest information" refers to data about things and themes that users are currently interested in, and is used to generate music and explanatory text.

[0175] "User location information" refers to data about the geographical location where the user is currently located, and is used to provide information about the surrounding area and appropriate content.

[0176] "User emotional information" refers to data that indicates the user's emotional state, analyzed from the user's facial expressions and tone of voice.

[0177] "Means for generating music and guidance text" refers to a process or apparatus for creating individually tailored music and guidance text based on the user's interest information, location information, and emotional information.

[0178] A "musical experience synchronized with the user's emotions" is an immersive entertainment experience created by combining music and explanatory text that are adjusted to the user's emotional state.

[0179] The system for realizing this invention consists of a user terminal, a server, and an emotion engine. The terminal is envisioned as smart glasses or a head-mounted display and is equipped with a camera and microphone to capture the user's facial expressions and voice tone. In particular, this emotion information is analyzed in real time and sent to the emotion engine.

[0180] The server is either run on specific hardware or located in the cloud. It houses modules for collecting user interest information, acquiring location information, and software necessary for analyzing sentiment information. Specific software includes OpenCV for image recognition and Google® Cloud Speech-to-Text for speech analysis.

[0181] Using a generative AI model, the server generates music and explanatory text. Examples of generative AI models include natural language generation engines such as GPT-3® or Bard. This allows the music's tone, tempo, and the content of the explanatory text to be adjusted according to the user's emotions. The generated content is then sent to the user's device.

[0182] As a concrete example of creating a musical experience that matches the user's emotions, consider a scenario where a user is using smart glasses while sightseeing. If the user's facial expression indicates enjoyment, the system plays calm and cheerful music and provides guidance such as, "There is a beautiful view nearby."

[0183] An example of a prompt from this system would be, "If the user's emotions are perceived as calm, generate calming guidance text about soothing music and scenery." This prompt allows the server to utilize its AI generation model to provide the user with the most optimal experience.

[0184] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0185] Step 1:

[0186] The device acquires the user's facial expressions and voice. Input consists of image data captured by the camera and voice data collected by the microphone. This data is sent to the emotion engine, which is then prepared for emotion analysis.

[0187] Step 2:

[0188] The emotion engine analyzes received facial expression data and voice data. It uses OpenCV for image processing and Google Cloud Speech-to-Text for voice analysis. The input is the user's facial expressions and voice tone, and the output is data indicating the user's emotional state. This analysis quantifies the emotions the user is currently experiencing.

[0189] Step 3:

[0190] The server collects user interest and location information. Inputs include user profile data and location information obtained from GPS sensors, while outputs are the information necessary for generating content based on interests and location.

[0191] Step 4:

[0192] The server generates music and guidance text using a generative AI model. The inputs here are emotional information, interest information, and location information. Based on this, prompt text is created and fed to the generative AI model (e.g., GPT-3) to output music and guidance text synchronized with the user's emotions.

[0193] Step 5:

[0194] The server combines the generated music and explanatory text to create a musical experience. The output is new content, including music and explanatory text that matches the emotional state, and is sent to the terminal.

[0195] Step 6:

[0196] The device provides the user with received musical experience content. The input is content sent from the server, and the output is what the user sees. The user enjoys a personalized experience combining music and explanatory text through smart glasses.

[0197] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0198] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0199] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0200] [Second Embodiment]

[0201] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0202] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0203] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0204] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0205] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0206] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0207] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0208] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0209] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0210] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0211] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0212] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0213] Embodiments of the present invention relate to a system that utilizes user interest information and location information to provide a personalized musical experience. The system mainly consists of a server, a terminal, and a user.

[0214] The server receives location and interest information transmitted from the user's smartphone or other device, and analyzes this data using a generative AI model. Based on the analyzed data, it is responsible for generating music content and navigation instructions that match the user's interests at that time. For example, if the user enjoys visiting cafes, it will create an original song about popular cafes in the area, along with corresponding directions.

[0215] The terminal plays musical content received from the server through the user interface. The combination of music and explanatory text allows users to enjoy learning while exploring the city. For example, a user visiting Shinjuku might receive a musical-style guide to Cafe Street through the terminal, followed by navigation to their next destination.

[0216] Users experience content provided through an application on their device. Following the provided music and guidance, users can increase their motivation to visit new places while also addressing a sedentary lifestyle. During this experience, users receive real-time information and contribute to improving future experiences by providing feedback.

[0217] As described above, the system of the present invention utilizes the user's location and individual interest data to provide a personalized and highly interactive experience, aiming to promote the alleviation of sedentary lifestyles in a gamified manner.

[0218] The following describes the processing flow.

[0219] Step 1:

[0220] The device uses GPS to obtain the user's current location when the application is launched, and extracts interest information from past search history and user profile. This data is encrypted and sent to the server.

[0221] Step 2:

[0222] The server receives location and interest information transmitted from the terminal. Using this information as input data, it starts content generation processing based on an AI model.

[0223] Step 3:

[0224] The server aggregates relevant geographical point and spot information based on the user's current location. Simultaneously, it generates music content and guide text tailored to the user's interests.

[0225] Step 4:

[0226] The server integrates the generated music and explanatory text to create a musical experience content for the user, and then sends that data to the terminal.

[0227] Step 5:

[0228] The device plays musical experience content received from the server. This allows the user to explore the city through sight and sound.

[0229] Step 6:

[0230] Users navigate the city following the content provided via their devices. They enjoy navigation to their destination accompanied by music through a musical experience.

[0231] Step 7:

[0232] The device collects user feedback and sends it to the server. The server analyzes this feedback and uses it to improve the generation algorithm, which is then reflected in future content generation.

[0233] (Example 1)

[0234] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0235] In modern society, there is a demand for information tailored to individual users. However, conventional music streaming and navigation systems have struggled to provide personalized experiences that match individual users' interests and location information. Furthermore, they lacked mechanisms to reflect user feedback in real time and continuously improve the experience. As a result, there is a problem of decreased user experience value and inability to increase satisfaction.

[0236] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0237] In this invention, the server includes means for acquiring user interest information, means for acquiring user location information, and means for generating prompt data using generative artificial intelligence based on the interest information and location information, and for generating music and guidance text. This makes it possible to provide each user with a personalized musical experience based on their interests and current location information.

[0238] A "user" refers to an individual who uses a system and experiences it.

[0239] "Interest information" refers to data about specific fields or activities that a user is interested in.

[0240] "Location information" refers to geographical data that indicates the user's current location.

[0241] "Generative artificial intelligence" refers to a computer system that generates information based on large datasets, using natural language processing and other techniques.

[0242] "Prompt data" refers to the input data used by generative artificial intelligence to generate information.

[0243] "Music" refers to auditory content generated for the user.

[0244] "Information text" refers to text data intended to provide navigation and information to users.

[0245] A "musical experience" refers to an interactive experience that combines music and explanatory text.

[0246] "User interface" refers to the screens and other means of interaction that users use when directly interacting with a system.

[0247] "Feedback" refers to information that users provide to the system to evaluate their experience and offer their opinions.

[0248] This system is designed to provide users with a more personalized musical experience. It primarily consists of servers, terminals, and users.

[0249] The server's role is to analyze user interest and location information using a generative AI model to generate personalized music and navigation text. The server is located in a cloud environment, and the generative AI model used is one that excels at natural language generation. For example, a large language model like GPT-4 could be considered. This generative AI model takes a prompt sentence as input and generates appropriate music content and navigation guidance. For example, a prompt sentence might be in the format of "The user is currently in Shinjuku and likes cafes. Please generate musical songs about nearby cafes."

[0250] The terminal plays the role of playing content sent from the server. Terminals include smartphones and tablet devices, and users receive music and navigation guidance through applications. By playing music on the user interface while displaying guidance text on the screen, it makes it easier for users to explore the city.

[0251] Users visit new locations by following music and instructions provided through their devices. This format encourages users to explore in an enjoyable and meaningful way, and allows for real-time feedback. User feedback is collected by the server to improve future experiences and helps refine the algorithms of the generative AI model.

[0252] This invention provides users with a more personalized and interest-based interactive experience. This allows us to meet the modern need for personalized information while simultaneously improving user satisfaction.

[0253] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0254] Step 1:

[0255] The server receives interest information and location information from the user's device. Based on this input data, it understands what the user is interested in and their current geographical location. Specifically, it utilizes GPS data provided by the device and interest information specified within the app, and aggregates this information on the server. The server's output is the interest information and location information for each user, which is necessary for analysis.

[0256] Step 2:

[0257] The server generates a prompt based on the received interest and location information. This prompt is then formatted as data to be input to the generative AI model. Specifically, it creates a sentence in a format easily understood by the generative AI model, such as, "The user is currently in △△ and likes □□. Please generate musical songs about the surrounding □□." The server's output is this prompt.

[0258] Step 3:

[0259] The server inputs the generated prompt text into a generation AI model to generate music and navigation text. Based on the prompt text provided as input, the generation AI model uses its built-in algorithms to generate music content and navigation guidance that corresponds to the user's interests. For example, if the user is assumed to like cafes, music and directions related to nearby cafes will be generated. The server's output consists of the generated music data and navigation text.

[0260] Step 4:

[0261] The server sends the generated music content and navigation instructions to the terminal. During this process, the data is encrypted to ensure secure transmission. The terminal then appropriately analyzes the data received from the server and prepares it for display as output on the user interface.

[0262] Step 5:

[0263] The device plays received music content and displays navigation instructions. Specifically, while the user explores the city, the device plays music through earphones or speakers, and displays the next destination and route information on the screen. The user can then act accordingly.

[0264] Step 6:

[0265] Users explore and experience the provided content, providing feedback on their devices. Users evaluate the quality of the experience and the accuracy of the guidance, and the device sends this data to the server. This feedback is used to improve the algorithm of the next generation AI model. The output is feedback data.

[0266] (Application Example 1)

[0267] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0268] In modern society, users are expected to efficiently acquire information that matches their preferences and enjoy fulfilling experiences. However, existing information provision systems struggle to provide flexible guidance tailored to individual interests and to deliver personalized, immersive experiences. In particular, in spaces visited by users with diverse interests, there is a need to provide new experiences that integrate music and navigation based on individual preferences.

[0269] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0270] In this invention, the server includes means for acquiring user preference information, means for acquiring user location data, and means for generating music and guidance information based on the preference information and location data. This makes it possible for users to enjoy a personalized musical experience in which they can enjoy music and guidance information tailored to their individual interests.

[0271] "User preference information" refers to information about the interests and concerns of individual users, specifically the distinctive preferences they show regarding particular music genres or places.

[0272] "Location data" refers to geographical information about the user's current location or the location they are planning to move to, and is obtained using GPS or other location measurement technologies.

[0273] "Means for generating music and navigation information" refers to a system that has the function of creating music content and navigation information optimized for the user based on collected user preference information and location data.

[0274] "Means of constructing a musical experience" refers to methods and processes for integrating generated music and informational data to create a new sensory experience for users.

[0275] "Means of delivering a musical experience" refers to the system for actually delivering a constructed musical experience to users, and is usually provided to users through terminals or devices.

[0276] "A spatial route designed based on user interests" refers to a spatial design that takes into account the user's preferences and location data to create a movement path optimized for that individual.

[0277] "Methods for generating custom music" refers to technologies that create music individually tailored to specific situations and interests, based on information obtained from the user's preferences and location.

[0278] In embodiments of the present invention, the entire system consists of the user's smart device and a server. The server acquires location data and preference information transmitted from the user's device and generates music and guidance information using a generative AI model based on this data. The generated content is transmitted to the user's device and provided there as a musical experience.

[0279] As hardware, the user's smartphone or mobile information terminal is used, and these devices acquire location information. Also, a server for operating the generative AI model operates on cloud services. Specific software used includes Python for data analysis and OpenAI, a machine learning library as a generative AI model.

[0280] As a specific example, when a user visits a large shopping center, information on stores that the user is interested in within the shopping center is aggregated, and based on that information, a musical guide dedicated to the user is generated. This guide combines music that suits the user's interests and store guides to support a more enjoyable and efficient exploration.

[0281] As an example of a prompt sentence, "Generate an original musical guide based on the category of interest and design the most appropriate in-store route based on the specified coordinates. Interests: {interest information}, Current location: {location information}" is used. By inputting this prompt sentence into the generative AI model, a mechanism is provided to provide the most suitable content for the user in real time.

[0282] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0283] Step 1:

[0284] The user's device acquires the current location. The user's location data is collected using GPS and sent to the server. The input is location coordinate data, and the output is the location information sent to the server. As a specific operation, the location information service inside the terminal acquires the location in the background, updates it in real time, and continuously sends it to the server.

[0285] Step 2:

[0286] The server obtains the user's preference information. The server obtains individual preference data from the user profile or the selected interest category. The input is the user's interest information, and the output is an interest dataset. As an operation, based on the data regularly transmitted from the device, the server individually evaluates the user's preferences and accesses the database in the server to obtain interest information.

[0287] Step 3:

[0288] The server utilizes the generated AI model with location data and preference information to generate music and guidance information. The input is location data and preference information, and the output is personalized music content and navigation information. The generated AI model on the server analyzes this input data and generates appropriate content based on the prompt text.

[0289] Step 4:

[0290] The server transmits the content generated to the user's terminal. The server transfers data packets containing music and guidance information to the user's device. The output is the data packet received by the terminal. The data transmitted from the server is efficiently packetized according to the network protocol and delivered to the terminal.

[0291] Step 5:

[0292] The user's terminal plays the received music and guidance information as a musical experience. The input is the data packet transmitted from the server, and the output is the music and guidance received by the user through hearing and vision. The terminal plays the music with an audio player and displays the navigation information on the screen.

[0293] Step 6:

[0294] Users enter feedback on the experience provided. This feedback is collected and sent to the server. The input is user feedback, and the output is feedback data used for future generation. Users enter their opinions into forms or questionnaires displayed on their devices, and upon completion, this information is registered in the database.

[0295] Step 7:

[0296] The server utilizes user feedback to improve its generation algorithm. The input is feedback data, and the output is the improved algorithm. By analyzing the feedback information collected on the server and updating the training dataset for the generation AI model, the user can further personalize their experience in subsequent uses.

[0297] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0298] Embodiments of the present invention relate to a system that provides a more personalized entertainment experience by offering a musical experience adapted to the user's emotional state. This system includes terminal, server, and user components, and further incorporates an emotion engine that performs emotion recognition.

[0299] The device acquires emotional data such as the user's facial expressions and voice tone. This data is analyzed in real time within the device or through a connected emotion engine. The results are sent to the server as the user's emotional state. This emotional information, along with other interest information and location information, is also sent to the server.

[0300] The server uses the received emotion information, interest information, and location information to drive a generative AI model to generate music and guiding texts suitable for the user. Additionally, the emotion engine utilizes the data analyzed from the user's emotions to dynamically adjust the tone, tempo of the music, and the content of the guiding texts, forming the most suitable content for the user's current situation. For example, if the user is in a relaxed state, it provides gentle music and calming guiding texts; when the user is energetic, it generates energetic music and inspiring guiding texts.

[0301] After that, the server transmits the adjusted musical experience content to the terminal. Through this content played on the terminal, the user can proactively enjoy a rich experience rooted in their emotions. The user can use a smartphone to engage in daily activities and tourism while enjoying music synchronized with their emotions, and furthermore, be stimulated by guiding texts that are changed in real-time.

[0302] In this way, the present invention opens up a new path to make eliminating lack of exercise enjoyable while maintaining emotional engagement according to the user's individual state.

[0303] The processing flow will be described below.

[0304] Step 1:

[0305] The terminal obtains emotion data by obtaining the user's permission to use the smartphone's camera and microphone to capture the user's expressions and voice. The emotion engine analyzes this data in real-time to identify the user's emotional state.

[0306] Step 2:

[0307] The terminal transmits the obtained emotional state data, along with the user's interest information and location information, to the server. These pieces of information serve as the basic data for successfully constructing the user's personalized experience.

[0308] Step 3:

[0309] The server comprehensively analyzes received emotional, interest, and location information. It then uses an AI model to generate music content and guidance text best suited to the user's current situation. During this process, the atmosphere of the music and the tone of the guidance text are dynamically adjusted according to the user's emotional state.

[0310] Step 4:

[0311] The server sends the generated, personalized musical experience content to the device. This content is optimized based on the user's past history and current emotions.

[0312] Step 5:

[0313] The device plays content received from the server, providing users with a musical experience. Users can explore the city while on the go through music and guidance synchronized with their emotions, and enjoy real-time updated information.

[0314] Step 6:

[0315] Users can provide feedback based on their experience through their device. The device sends this feedback to the server. The server analyzes the collected feedback and uses it to improve algorithms that further enhance the user experience in future content creation.

[0316] (Example 2)

[0317] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0318] Modern entertainment systems face challenges in providing individually customized content based on a user's emotional state, potentially leading to decreased user satisfaction. Furthermore, there is a lack of systems that effectively utilize user interest and location information to deliver optimal entertainment in real time. This invention aims to improve the personalized entertainment experience by considering the user's emotional state and generating more individualized voice and guidance text.

[0319] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0320] In this invention, the server includes means for acquiring user emotional data, means for analyzing the acquired emotional data, and means for acquiring user interest information. This makes it possible to personalize voice and guidance text based on the user's emotional state and provide a user-optimized entertainment experience in real time.

[0321] "Emotional data" refers to information that indicates a user's emotional state, obtained from things like their facial expressions and tone of voice.

[0322] "Analysis" refers to the process of processing acquired emotional data to identify or classify the user's emotions.

[0323] "Interest information" refers to information about a user's interests and preferences, and is data used to generate personalized entertainment experiences.

[0324] "Location information" is data that indicates the user's current geographical location and is used to provide content based on the user's movement and surrounding environment.

[0325] "Sound" refers to music and sounds generated based on the user's emotional state, and is an element that enriches the user's experience.

[0326] An "informative text" is a text or audio message associated with audio that provides information to the user.

[0327] An "entertainment experience" is a series of user experiences that combine customized audio and guidance text generated based on emotional data, interest information, and location information.

[0328] "Real-time" refers to a process where information processing and content generation are performed in response to the user's current state.

[0329] This invention is a system that provides personalized entertainment experiences based on the user's emotional state. The system consists of a terminal, a server, and a user, and utilizes the user's emotions, interests, and location information to generate optimal voice and guidance text.

[0330] The terminal is a device equipped with a camera and microphone as a means of acquiring user emotional data. Specifically, it can utilize common electronic devices such as smartphones and tablets. The terminal uses the sensors of these devices to collect the user's facial expressions and voice tone in real time and analyzes them using an emotion engine. This analysis makes it possible to understand the user's emotional state.

[0331] The server receives emotional information, interest information, and location information transmitted from the terminal, and combines this data to generate voice and guidance text using a generative AI model. The generative AI model can utilize, for example, open-source natural language processing models or commercial AI services. This makes it possible to adjust the voice tone and guidance text content according to the user's emotions.

[0332] Users utilize this system while carrying their devices and moving around. For example, by going out with their smartphone and walking around, they can receive music that matches their mood and real-time guidance. This allows users to enjoy a richer experience. An example of a specific prompt message is sending the instruction, "Generate calming music suitable for when the user is relaxed," to the generation AI model. Based on such prompt messages, the system can provide optimal content that meets the user's needs.

[0333] Thus, this system is designed to provide a dynamic and personalized entertainment experience that takes into account the user's emotions and activities.

[0334] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0335] Step 1:

[0336] The device uses a camera and microphone to acquire emotional data such as the user's facial expressions and voice tone. The input is the user's physical facial expressions and voice, while the output is digitized emotional data. This data is collected in real time for use in subsequent analysis steps.

[0337] Step 2:

[0338] The device analyzes the acquired emotional data using an emotion engine. This analysis uses a machine learning model to evaluate facial feature points and voice frequency components to identify the emotional state. The input is the emotional data obtained in step 1, and the output is an emotion label such as "happy," "sad," or "surprised." This quantifies the user's emotional state.

[0339] Step 3:

[0340] The device collects user interest information and location information along with emotional information, and sends them to the server. The input for this step is the analyzed emotional information, pre-configured interest information, and location information obtained via GPS, etc., and the output is the data to be sent to the server. The data is transmitted securely through a secure communication method.

[0341] Step 4:

[0342] The server uses the received emotion, interest, and location information to send prompt messages to the generating AI model. This input is the combined information obtained in step 3, and the model generates voice and guidance text. The output is an audio file and guidance text optimized for the user's situation. This process creates personalized content based on the user's emotions.

[0343] Step 5:

[0344] The server sends the generated audio and guidance text to the terminal. The input for this step is the generated content obtained in step 4, and the output is its transfer to the terminal. The terminal receives this transmitted data and begins playing it at a time appropriate for the user.

[0345] Step 6:

[0346] The terminal plays the transmitted audio and guidance text, providing the user with an entertainment experience. The input is the audio file and guidance text received in step 5, and the output is the audiovisual content that the user actually experiences. In this step, the playback content may be dynamically adjusted in response to the user's reactions and the environment while moving.

[0347] (Application Example 2)

[0348] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0349] Traditional entertainment experiences have struggled to provide content tailored to individual user emotions. As a result, users are often provided with experiences that are not necessarily optimal, failing to fully realize the appeal of entertainment. In particular, virtual environments often lack a sense of presence and immersion, making it essential to improve user engagement by providing personalized experiences.

[0350] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0351] In this invention, the server includes means for acquiring user interest information, means for acquiring user location information, means for analyzing user emotional information, and means for generating music and guidance text based on the interest information, location information, and emotional information. This enables a personalized musical experience synchronized with the user's emotions, enhancing realism and immersion, and providing a more individualized and optimized entertainment experience.

[0352] "User interest information" refers to data about things and themes that users are currently interested in, and is used to generate music and explanatory text.

[0353] "User location information" refers to data about the geographical location where the user is currently located, and is used to provide information about the surrounding area and appropriate content.

[0354] "User emotional information" refers to data that indicates the user's emotional state, analyzed from the user's facial expressions and tone of voice.

[0355] "Means for generating music and guidance text" refers to a process or apparatus for creating individually tailored music and guidance text based on the user's interest information, location information, and emotional information.

[0356] A "musical experience synchronized with the user's emotions" is an immersive entertainment experience created by combining music and explanatory text that are adjusted to the user's emotional state.

[0357] The system for realizing this invention consists of a user terminal, a server, and an emotion engine. The terminal is envisioned as smart glasses or a head-mounted display and is equipped with a camera and microphone to capture the user's facial expressions and voice tone. In particular, this emotion information is analyzed in real time and sent to the emotion engine.

[0358] The server is either run on specific hardware or located in the cloud. It houses modules for collecting user interest information, acquiring location information, and software necessary for analyzing sentiment information. Specific software includes OpenCV for image recognition and Google Cloud Speech-to-Text for speech analysis.

[0359] Using a generative AI model, the server generates music and explanatory text. Examples of generative AI models include natural language generation engines such as GPT-3 and Bard. This allows the music's tone, tempo, and the content of the explanatory text to be adjusted according to the user's emotions. The generated content is then sent to the user's device.

[0360] As a concrete example of creating a musical experience that matches the user's emotions, consider a scenario where a user is using smart glasses while sightseeing. If the user's facial expression indicates enjoyment, the system plays calm and cheerful music and provides guidance such as, "There is a beautiful view nearby."

[0361] An example of a prompt from this system would be, "If the user's emotions are perceived as calm, generate calming guidance text about soothing music and scenery." This prompt allows the server to utilize its AI generation model to provide the user with the most optimal experience.

[0362] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0363] Step 1:

[0364] The device acquires the user's facial expressions and voice. Input consists of image data captured by the camera and voice data collected by the microphone. This data is sent to the emotion engine, which is then prepared for emotion analysis.

[0365] Step 2:

[0366] The emotion engine analyzes received facial expression data and voice data. It uses OpenCV for image processing and Google Cloud Speech-to-Text for voice analysis. The input is the user's facial expressions and voice tone, and the output is data indicating the user's emotional state. This analysis quantifies the emotions the user is currently experiencing.

[0367] Step 3:

[0368] The server collects user interest and location information. Inputs include user profile data and location information obtained from GPS sensors, while outputs are the information necessary for generating content based on interests and location.

[0369] Step 4:

[0370] The server generates music and guidance text using a generative AI model. The inputs here are emotional information, interest information, and location information. Based on this, prompt text is created and fed to the generative AI model (e.g., GPT-3) to output music and guidance text synchronized with the user's emotions.

[0371] Step 5:

[0372] The server combines the generated music and explanatory text to create a musical experience. The output is new content, including music and explanatory text that matches the emotional state, and is sent to the terminal.

[0373] Step 6:

[0374] The device provides the user with received musical experience content. The input is content sent from the server, and the output is what the user sees. The user enjoys a personalized experience combining music and explanatory text through smart glasses.

[0375] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0376] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0377] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0378] [Third Embodiment]

[0379] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0380] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0381] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0382] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0383] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0384] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0385] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0386] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0387] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0388] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0389] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0390] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0391] Embodiments of the present invention relate to a system that utilizes user interest information and location information to provide a personalized musical experience. The system mainly consists of a server, a terminal, and a user.

[0392] The server receives location and interest information transmitted from the user's smartphone or other device, and analyzes this data using a generative AI model. Based on the analyzed data, it is responsible for generating music content and navigation instructions that match the user's interests at that time. For example, if the user enjoys visiting cafes, it will create an original song about popular cafes in the area, along with corresponding directions.

[0393] The terminal plays musical content received from the server through the user interface. The combination of music and explanatory text allows users to enjoy learning while exploring the city. For example, a user visiting Shinjuku might receive a musical-style guide to Cafe Street through the terminal, followed by navigation to their next destination.

[0394] Users experience content provided through an application on their device. Following the provided music and guidance, users can increase their motivation to visit new places while also addressing a sedentary lifestyle. During this experience, users receive real-time information and contribute to improving future experiences by providing feedback.

[0395] As described above, the system of the present invention utilizes the user's location and individual interest data to provide a personalized and highly interactive experience, aiming to promote the alleviation of sedentary lifestyles in a gamified manner.

[0396] The following describes the processing flow.

[0397] Step 1:

[0398] The device uses GPS to obtain the user's current location when the application is launched, and extracts interest information from past search history and user profile. This data is encrypted and sent to the server.

[0399] Step 2:

[0400] The server receives location and interest information transmitted from the terminal. Using this information as input data, it starts content generation processing based on an AI model.

[0401] Step 3:

[0402] The server aggregates relevant geographical point and spot information based on the user's current location. Simultaneously, it generates music content and guide text tailored to the user's interests.

[0403] Step 4:

[0404] The server integrates the generated music and explanatory text to create a musical experience content for the user, and then sends that data to the terminal.

[0405] Step 5:

[0406] The device plays musical experience content received from the server. This allows the user to explore the city through sight and sound.

[0407] Step 6:

[0408] Users navigate the city following the content provided via their devices. They enjoy navigation to their destination accompanied by music through a musical experience.

[0409] Step 7:

[0410] The device collects user feedback and sends it to the server. The server analyzes this feedback and uses it to improve the generation algorithm, which is then reflected in future content generation.

[0411] (Example 1)

[0412] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0413] In modern society, there is a demand for information tailored to individual users. However, conventional music streaming and navigation systems have struggled to provide personalized experiences that match individual users' interests and location information. Furthermore, they lacked mechanisms to reflect user feedback in real time and continuously improve the experience. As a result, there is a problem of decreased user experience value and inability to increase satisfaction.

[0414] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0415] In this invention, the server includes means for acquiring user interest information, means for acquiring user location information, and means for generating prompt data using generative artificial intelligence based on the interest information and location information, and for generating music and guidance text. This makes it possible to provide each user with a personalized musical experience based on their interests and current location information.

[0416] A "user" refers to an individual who uses a system and experiences it.

[0417] "Interest information" refers to data about specific fields or activities that a user is interested in.

[0418] "Location information" refers to geographical data that indicates the user's current location.

[0419] "Generative artificial intelligence" refers to a computer system that generates information based on large datasets, using natural language processing and other techniques.

[0420] "Prompt data" refers to the input data used by generative artificial intelligence to generate information.

[0421] "Music" refers to auditory content generated for the user.

[0422] "Information text" refers to text data intended to provide navigation and information to users.

[0423] A "musical experience" refers to an interactive experience that combines music and explanatory text.

[0424] "User interface" refers to the screens and other means of interaction that users use when directly interacting with a system.

[0425] "Feedback" refers to information that users provide to the system to evaluate their experience and offer their opinions.

[0426] This system is designed to provide users with a more personalized musical experience. It primarily consists of servers, terminals, and users.

[0427] The server's role is to analyze user interest and location information using a generative AI model to generate personalized music and navigation text. The server is located in a cloud environment, and the generative AI model used is one that excels at natural language generation. For example, a large language model like GPT-4 could be considered. This generative AI model takes a prompt sentence as input and generates appropriate music content and navigation guidance. For example, a prompt sentence might be in the format of "The user is currently in Shinjuku and likes cafes. Please generate musical songs about nearby cafes."

[0428] The terminal plays the role of playing content sent from the server. Terminals include smartphones and tablet devices, and users receive music and navigation guidance through applications. By playing music on the user interface while displaying guidance text on the screen, it makes it easier for users to explore the city.

[0429] Users visit new locations by following music and instructions provided through their devices. This format encourages users to explore in an enjoyable and meaningful way, and allows for real-time feedback. User feedback is collected by the server to improve future experiences and helps refine the algorithms of the generative AI model.

[0430] This invention provides users with a more personalized and interest-based interactive experience. This allows us to meet the modern need for personalized information while simultaneously improving user satisfaction.

[0431] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0432] Step 1:

[0433] The server receives interest information and location information from the user's device. Based on this input data, it understands what the user is interested in and their current geographical location. Specifically, it utilizes GPS data provided by the device and interest information specified within the app, and aggregates this information on the server. The server's output is the interest information and location information for each user, which is necessary for analysis.

[0434] Step 2:

[0435] The server generates a prompt based on the received interest and location information. This prompt is then formatted as data to be input to the generative AI model. Specifically, it creates a sentence in a format easily understood by the generative AI model, such as, "The user is currently in △△ and likes □□. Please generate musical songs about the surrounding □□." The server's output is this prompt.

[0436] Step 3:

[0437] The server inputs the generated prompt text into a generation AI model to generate music and navigation text. Based on the prompt text provided as input, the generation AI model uses its built-in algorithms to generate music content and navigation guidance that corresponds to the user's interests. For example, if the user is assumed to like cafes, music and directions related to nearby cafes will be generated. The server's output consists of the generated music data and navigation text.

[0438] Step 4:

[0439] The server sends the generated music content and navigation instructions to the terminal. During this process, the data is encrypted to ensure secure transmission. The terminal then appropriately analyzes the data received from the server and prepares it for display as output on the user interface.

[0440] Step 5:

[0441] The device plays received music content and displays navigation instructions. Specifically, while the user explores the city, the device plays music through earphones or speakers, and displays the next destination and route information on the screen. The user can then act accordingly.

[0442] Step 6:

[0443] Users explore and experience the provided content, providing feedback on their devices. Users evaluate the quality of the experience and the accuracy of the guidance, and the device sends this data to the server. This feedback is used to improve the algorithm of the next generation AI model. The output is feedback data.

[0444] (Application Example 1)

[0445] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0446] In modern society, users are expected to efficiently acquire information that matches their preferences and enjoy fulfilling experiences. However, existing information provision systems struggle to provide flexible guidance tailored to individual interests and to deliver personalized, immersive experiences. In particular, in spaces visited by users with diverse interests, there is a need to provide new experiences that integrate music and navigation based on individual preferences.

[0447] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0448] In this invention, the server includes means for acquiring user preference information, means for acquiring user location data, and means for generating music and guidance information based on the preference information and location data. This makes it possible for users to enjoy a personalized musical experience in which they can enjoy music and guidance information tailored to their individual interests.

[0449] "User preference information" refers to information about the interests and concerns of individual users, specifically the distinctive preferences they show regarding particular music genres or places.

[0450] "Location data" refers to geographical information about the user's current location or the location they are planning to move to, and is obtained using GPS or other location measurement technologies.

[0451] "Means for generating music and navigation information" refers to a system that has the function of creating music content and navigation information optimized for the user based on collected user preference information and location data.

[0452] "Means of constructing a musical experience" refers to methods and processes for integrating generated music and informational data to create a new sensory experience for users.

[0453] "Means of delivering a musical experience" refers to the system for actually delivering a constructed musical experience to users, and is usually provided to users through terminals or devices.

[0454] "A spatial route designed based on user interests" refers to a spatial design that takes into account the user's preferences and location data to create a movement path optimized for that individual.

[0455] "Methods for generating custom music" refers to technologies that create music individually tailored to specific situations and interests, based on information obtained from the user's preferences and location.

[0456] In embodiments of the present invention, the entire system consists of the user's smart device and a server. The server acquires location data and preference information transmitted from the user's device and generates music and guidance information using a generative AI model based on this data. The generated content is transmitted to the user's device and provided there as a musical experience.

[0457] The hardware used will be the user's smartphone or mobile device, which will acquire location information. Additionally, the server running the generative AI model will operate on a cloud service. Specific software used will include Python for data analysis and OpenAI, a machine learning library for generating AI models.

[0458] As a concrete example, when a user visits a large shopping center, information on stores that the user is interested in is compiled, and based on that information, a personalized musical guide is generated for the user. This guide combines music that matches the user's interests with store information, supporting a more enjoyable and efficient exploration.

[0459] An example of a prompt message is: "Generate an original musical guide based on the user's interests and design the most appropriate in-store route based on the specified coordinates. Interests: {interest information}, Current location: {location information}". By inputting this prompt message into the AI ​​model, the system provides the user with the most suitable content in real time.

[0460] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0461] Step 1:

[0462] The user's device obtains its current location. It uses GPS to collect the user's location data and sends it to the server. The input is location coordinate data, and the output is location information sent to the server. Specifically, the device's internal location service obtains the location in the background, updates it in real time, and continuously sends it to the server.

[0463] Step 2:

[0464] The server retrieves user preference information. The server obtains individual preference data from the user profile or selected interest categories. The input is user interest information, and the output is an interest dataset. The process involves individually evaluating user preferences based on data periodically sent from the device, and accessing a database on the server to retrieve interest information.

[0465] Step 3:

[0466] The server uses location data and preference information to generate music and navigation information using a generative AI model. The input is location data and preference information, and the output is personalized music content and navigation information. The generative AI model on the server analyzes this input data and generates appropriate content through prompts.

[0467] Step 4:

[0468] The server sends the generated content to the user's terminal. The server transfers data packets containing music and guidance information to the user's device. The output is the data packets received by the terminal. The data transmitted from the server is efficiently packetized according to the network protocol and delivered to the terminal.

[0469] Step 5:

[0470] The system plays music and navigation information received by the user's device as a musical experience. Input is data packets sent from the server, and output is music and navigation received by the user aurally and visually. The device plays the music using an audio player and displays navigation information on the screen.

[0471] Step 6:

[0472] Users enter feedback on the experience provided. This feedback is collected and sent to the server. The input is user feedback, and the output is feedback data used for future generation. Users enter their opinions into forms or questionnaires displayed on their devices, and upon completion, this information is registered in the database.

[0473] Step 7:

[0474] The server utilizes user feedback to improve its generation algorithm. The input is feedback data, and the output is the improved algorithm. By analyzing the feedback information collected on the server and updating the training dataset for the generation AI model, the user can further personalize their experience in subsequent uses.

[0475] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0476] Embodiments of the present invention relate to a system that provides a more personalized entertainment experience by offering a musical experience adapted to the user's emotional state. This system includes terminal, server, and user components, and further incorporates an emotion engine that performs emotion recognition.

[0477] The device acquires emotional data such as the user's facial expressions and voice tone. This data is analyzed in real time within the device or through a connected emotion engine. The results are sent to the server as the user's emotional state. This emotional information, along with other interest information and location information, is also sent to the server.

[0478] The server uses a generative AI model to generate music and guidance text tailored to the user, based on received emotional, interest, and location information. Furthermore, the emotional engine uses data analyzing the user's emotions to dynamically adjust the music's tone and tempo, as well as the content of the guidance text, creating content that best suits the user's current state. For example, if the user is relaxed, it provides calm music and soothing guidance text; if they are energetic, it generates energetic music and dynamic guidance text.

[0479] The server then sends the tailored musical experience content to the device. Through this content, played on the device, users can proactively enjoy a rich experience rooted in their emotions. Users can enjoy music synchronized with their emotions while using their smartphones for everyday commutes and sightseeing, and their interest is further piqued by real-time changing guided text.

[0480] Thus, the present invention opens a new path to making the elimination of sedentary lifestyle enjoyable, while maintaining emotional engagement tailored to the individual user's situation.

[0481] The following describes the processing flow.

[0482] Step 1:

[0483] The device, with the user's permission, uses the smartphone's camera and microphone to capture the user's facial expressions and voice, thereby acquiring emotional data. The emotion engine analyzes this data in real time to identify the user's emotional state.

[0484] Step 2:

[0485] The device transmits user interest information and location information to the server along with acquired emotional state data. This information serves as foundational data for building a personalized user experience.

[0486] Step 3:

[0487] The server comprehensively analyzes received emotional, interest, and location information. It then uses an AI model to generate music content and guidance text best suited to the user's current situation. During this process, the atmosphere of the music and the tone of the guidance text are dynamically adjusted according to the user's emotional state.

[0488] Step 4:

[0489] The server sends the generated, personalized musical experience content to the device. This content is optimized based on the user's past history and current emotions.

[0490] Step 5:

[0491] The device plays content received from the server, providing users with a musical experience. Users can explore the city while on the go through music and guidance synchronized with their emotions, and enjoy real-time updated information.

[0492] Step 6:

[0493] Users can provide feedback based on their experience through their device. The device sends this feedback to the server. The server analyzes the collected feedback and uses it to improve algorithms that further enhance the user experience in future content creation.

[0494] (Example 2)

[0495] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0496] Modern entertainment systems face challenges in providing individually customized content based on a user's emotional state, potentially leading to decreased user satisfaction. Furthermore, there is a lack of systems that effectively utilize user interest and location information to deliver optimal entertainment in real time. This invention aims to improve the personalized entertainment experience by considering the user's emotional state and generating more individualized voice and guidance text.

[0497] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0498] In this invention, the server includes means for acquiring user emotional data, means for analyzing the acquired emotional data, and means for acquiring user interest information. This makes it possible to personalize voice and guidance text based on the user's emotional state and provide a user-optimized entertainment experience in real time.

[0499] "Emotional data" refers to information that indicates a user's emotional state, obtained from things like their facial expressions and tone of voice.

[0500] "Analysis" refers to the process of processing acquired emotional data to identify or classify the user's emotions.

[0501] "Interest information" refers to information about a user's interests and preferences, and is data used to generate personalized entertainment experiences.

[0502] "Location information" is data that indicates the user's current geographical location and is used to provide content based on the user's movement and surrounding environment.

[0503] "Sound" refers to music and sounds generated based on the user's emotional state, and is an element that enriches the user's experience.

[0504] An "informative text" is a text or audio message associated with audio that provides information to the user.

[0505] An "entertainment experience" is a series of user experiences that combine customized audio and guidance text generated based on emotional data, interest information, and location information.

[0506] "Real-time" refers to a process where information processing and content generation are performed in response to the user's current state.

[0507] This invention is a system that provides personalized entertainment experiences based on the user's emotional state. The system consists of a terminal, a server, and a user, and utilizes the user's emotions, interests, and location information to generate optimal voice and guidance text.

[0508] The terminal is a device equipped with a camera and microphone as a means of acquiring user emotional data. Specifically, it can utilize common electronic devices such as smartphones and tablets. The terminal uses the sensors of these devices to collect the user's facial expressions and voice tone in real time and analyzes them using an emotion engine. This analysis makes it possible to understand the user's emotional state.

[0509] The server receives emotional information, interest information, and location information transmitted from the terminal, and combines this data to generate voice and guidance text using a generative AI model. The generative AI model can utilize, for example, open-source natural language processing models or commercial AI services. This makes it possible to adjust the voice tone and guidance text content according to the user's emotions.

[0510] Users utilize this system while carrying their devices and moving around. For example, by going out with their smartphone and walking around, they can receive music that matches their mood and real-time guidance. This allows users to enjoy a richer experience. An example of a specific prompt message is sending the instruction, "Generate calming music suitable for when the user is relaxed," to the generation AI model. Based on such prompt messages, the system can provide optimal content that meets the user's needs.

[0511] Thus, this system is designed to provide a dynamic and personalized entertainment experience that takes into account the user's emotions and activities.

[0512] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0513] Step 1:

[0514] The device uses a camera and microphone to acquire emotional data such as the user's facial expressions and voice tone. The input is the user's physical facial expressions and voice, while the output is digitized emotional data. This data is collected in real time for use in subsequent analysis steps.

[0515] Step 2:

[0516] The device analyzes the acquired emotional data using an emotion engine. This analysis uses a machine learning model to evaluate facial feature points and voice frequency components to identify the emotional state. The input is the emotional data obtained in step 1, and the output is an emotion label such as "happy," "sad," or "surprised." This quantifies the user's emotional state.

[0517] Step 3:

[0518] The device collects user interest information and location information along with emotional information, and sends them to the server. The input for this step is the analyzed emotional information, pre-configured interest information, and location information obtained via GPS, etc., and the output is the data to be sent to the server. The data is transmitted securely through a secure communication method.

[0519] Step 4:

[0520] The server uses the received emotion, interest, and location information to send prompt messages to the generating AI model. This input is the combined information obtained in step 3, and the model generates voice and guidance text. The output is an audio file and guidance text optimized for the user's situation. This process creates personalized content based on the user's emotions.

[0521] Step 5:

[0522] The server sends the generated audio and guidance text to the terminal. The input for this step is the generated content obtained in step 4, and the output is its transfer to the terminal. The terminal receives this transmitted data and begins playing it at a time appropriate for the user.

[0523] Step 6:

[0524] The terminal plays the transmitted audio and guidance text, providing the user with an entertainment experience. The input is the audio file and guidance text received in step 5, and the output is the audiovisual content that the user actually experiences. In this step, the playback content may be dynamically adjusted in response to the user's reactions and the environment while moving.

[0525] (Application Example 2)

[0526] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0527] Traditional entertainment experiences have struggled to provide content tailored to individual user emotions. As a result, users are often provided with experiences that are not necessarily optimal, failing to fully realize the appeal of entertainment. In particular, virtual environments often lack a sense of presence and immersion, making it essential to improve user engagement by providing personalized experiences.

[0528] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0529] In this invention, the server includes means for acquiring user interest information, means for acquiring user location information, means for analyzing user emotional information, and means for generating music and guidance text based on the interest information, location information, and emotional information. This enables a personalized musical experience synchronized with the user's emotions, enhancing realism and immersion, and providing a more individualized and optimized entertainment experience.

[0530] "User interest information" refers to data about things and themes that users are currently interested in, and is used to generate music and explanatory text.

[0531] "User location information" refers to data about the geographical location where the user is currently located, and is used to provide information about the surrounding area and appropriate content.

[0532] "User emotional information" refers to data that indicates the user's emotional state, analyzed from the user's facial expressions and tone of voice.

[0533] "Means for generating music and guidance text" refers to a process or apparatus for creating individually tailored music and guidance text based on the user's interest information, location information, and emotional information.

[0534] A "musical experience synchronized with the user's emotions" is an immersive entertainment experience created by combining music and explanatory text that are adjusted to the user's emotional state.

[0535] The system for realizing this invention consists of a user terminal, a server, and an emotion engine. The terminal is envisioned as smart glasses or a head-mounted display and is equipped with a camera and microphone to capture the user's facial expressions and voice tone. In particular, this emotion information is analyzed in real time and sent to the emotion engine.

[0536] The server is either run on specific hardware or located in the cloud. It houses modules for collecting user interest information, acquiring location information, and software necessary for analyzing sentiment information. Specific software includes OpenCV for image recognition and Google Cloud Speech-to-Text for speech analysis.

[0537] Using a generative AI model, the server generates music and explanatory text. Examples of generative AI models include natural language generation engines such as GPT-3 and Bard. This allows the music's tone, tempo, and the content of the explanatory text to be adjusted according to the user's emotions. The generated content is then sent to the user's device.

[0538] As a concrete example of creating a musical experience that matches the user's emotions, consider a scenario where a user is using smart glasses while sightseeing. If the user's facial expression indicates enjoyment, the system plays calm and cheerful music and provides guidance such as, "There is a beautiful view nearby."

[0539] An example of a prompt from this system would be, "If the user's emotions are perceived as calm, generate calming guidance text about soothing music and scenery." This prompt allows the server to utilize its AI generation model to provide the user with the most optimal experience.

[0540] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0541] Step 1:

[0542] The device acquires the user's facial expressions and voice. Input consists of image data captured by the camera and voice data collected by the microphone. This data is sent to the emotion engine, which is then prepared for emotion analysis.

[0543] Step 2:

[0544] The emotion engine analyzes received facial expression data and voice data. It uses OpenCV for image processing and Google Cloud Speech-to-Text for voice analysis. The input is the user's facial expressions and voice tone, and the output is data indicating the user's emotional state. This analysis quantifies the emotions the user is currently experiencing.

[0545] Step 3:

[0546] The server collects user interest and location information. Inputs include user profile data and location information obtained from GPS sensors, while outputs are the information necessary for generating content based on interests and location.

[0547] Step 4:

[0548] The server generates music and guidance text using a generative AI model. The inputs here are emotional information, interest information, and location information. Based on this, prompt text is created and fed to the generative AI model (e.g., GPT-3) to output music and guidance text synchronized with the user's emotions.

[0549] Step 5:

[0550] The server combines the generated music and explanatory text to create a musical experience. The output is new content, including music and explanatory text that matches the emotional state, and is sent to the terminal.

[0551] Step 6:

[0552] The device provides the user with received musical experience content. The input is content sent from the server, and the output is what the user sees. The user enjoys a personalized experience combining music and explanatory text through smart glasses.

[0553] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0554] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0555] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0556] [Fourth Embodiment]

[0557] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0558] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0559] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0560] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0561] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0562] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0563] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0564] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0565] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0566] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0567] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0568] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0569] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0570] Embodiments of the present invention relate to a system that utilizes user interest information and location information to provide a personalized musical experience. The system mainly consists of a server, a terminal, and a user.

[0571] The server receives location and interest information transmitted from the user's smartphone or other device, and analyzes this data using a generative AI model. Based on the analyzed data, it is responsible for generating music content and navigation instructions that match the user's interests at that time. For example, if the user enjoys visiting cafes, it will create an original song about popular cafes in the area, along with corresponding directions.

[0572] The terminal plays musical content received from the server through the user interface. The combination of music and explanatory text allows users to enjoy learning while exploring the city. For example, a user visiting Shinjuku might receive a musical-style guide to Cafe Street through the terminal, followed by navigation to their next destination.

[0573] Users experience content provided through an application on their device. Following the provided music and guidance, users can increase their motivation to visit new places while also addressing a sedentary lifestyle. During this experience, users receive real-time information and contribute to improving future experiences by providing feedback.

[0574] As described above, the system of the present invention utilizes the user's location and individual interest data to provide a personalized and highly interactive experience, aiming to promote the alleviation of sedentary lifestyles in a gamified manner.

[0575] The following describes the processing flow.

[0576] Step 1:

[0577] The device uses GPS to obtain the user's current location when the application is launched, and extracts interest information from past search history and user profile. This data is encrypted and sent to the server.

[0578] Step 2:

[0579] The server receives location and interest information transmitted from the terminal. Using this information as input data, it starts content generation processing based on an AI model.

[0580] Step 3:

[0581] The server aggregates relevant geographical point and spot information based on the user's current location. Simultaneously, it generates music content and guide text tailored to the user's interests.

[0582] Step 4:

[0583] The server integrates the generated music and explanatory text to create a musical experience content for the user, and then sends that data to the terminal.

[0584] Step 5:

[0585] The device plays musical experience content received from the server. This allows the user to explore the city through sight and sound.

[0586] Step 6:

[0587] Users navigate the city following the content provided via their devices. They enjoy navigation to their destination accompanied by music through a musical experience.

[0588] Step 7:

[0589] The device collects user feedback and sends it to the server. The server analyzes this feedback and uses it to improve the generation algorithm, which is then reflected in future content generation.

[0590] (Example 1)

[0591] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0592] In modern society, there is a demand for information tailored to individual users. However, conventional music streaming and navigation systems have struggled to provide personalized experiences that match individual users' interests and location information. Furthermore, they lacked mechanisms to reflect user feedback in real time and continuously improve the experience. As a result, there is a problem of decreased user experience value and inability to increase satisfaction.

[0593] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0594] In this invention, the server includes means for acquiring user interest information, means for acquiring user location information, and means for generating prompt data using generative artificial intelligence based on the interest information and location information, and for generating music and guidance text. This makes it possible to provide each user with a personalized musical experience based on their interests and current location information.

[0595] A "user" refers to an individual who uses a system and experiences it.

[0596] "Interest information" refers to data about specific fields or activities that a user is interested in.

[0597] "Location information" refers to geographical data that indicates the user's current location.

[0598] "Generative artificial intelligence" refers to a computer system that generates information based on large datasets, using natural language processing and other techniques.

[0599] "Prompt data" refers to the input data used by generative artificial intelligence to generate information.

[0600] "Music" refers to auditory content generated for the user.

[0601] "Information text" refers to text data intended to provide navigation and information to users.

[0602] A "musical experience" refers to an interactive experience that combines music and explanatory text.

[0603] "User interface" refers to the screens and other means of interaction that users use when directly interacting with a system.

[0604] "Feedback" refers to information that users provide to the system to evaluate their experience and offer their opinions.

[0605] This system is designed to provide users with a more personalized musical experience. It primarily consists of servers, terminals, and users.

[0606] The server's role is to analyze user interest and location information using a generative AI model to generate personalized music and navigation text. The server is located in a cloud environment, and the generative AI model used is one that excels at natural language generation. For example, a large language model like GPT-4 could be considered. This generative AI model takes a prompt sentence as input and generates appropriate music content and navigation guidance. For example, a prompt sentence might be in the format of "The user is currently in Shinjuku and likes cafes. Please generate musical songs about nearby cafes."

[0607] The terminal plays the role of playing content sent from the server. Terminals include smartphones and tablet devices, and users receive music and navigation guidance through applications. By playing music on the user interface while displaying guidance text on the screen, it makes it easier for users to explore the city.

[0608] Users visit new locations by following music and instructions provided through their devices. This format encourages users to explore in an enjoyable and meaningful way, and allows for real-time feedback. User feedback is collected by the server to improve future experiences and helps refine the algorithms of the generative AI model.

[0609] This invention provides users with a more personalized and interest-based interactive experience. This allows us to meet the modern need for personalized information while simultaneously improving user satisfaction.

[0610] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0611] Step 1:

[0612] The server receives interest information and location information from the user's device. Based on this input data, it understands what the user is interested in and their current geographical location. Specifically, it utilizes GPS data provided by the device and interest information specified within the app, and aggregates this information on the server. The server's output is the interest information and location information for each user, which is necessary for analysis.

[0613] Step 2:

[0614] The server generates a prompt based on the received interest and location information. This prompt is then formatted as data to be input to the generative AI model. Specifically, it creates a sentence in a format easily understood by the generative AI model, such as, "The user is currently in △△ and likes □□. Please generate musical songs about the surrounding □□." The server's output is this prompt.

[0615] Step 3:

[0616] The server inputs the generated prompt text into a generation AI model to generate music and navigation text. Based on the prompt text provided as input, the generation AI model uses its built-in algorithms to generate music content and navigation guidance that corresponds to the user's interests. For example, if the user is assumed to like cafes, music and directions related to nearby cafes will be generated. The server's output consists of the generated music data and navigation text.

[0617] Step 4:

[0618] The server sends the generated music content and navigation instructions to the terminal. During this process, the data is encrypted to ensure secure transmission. The terminal then appropriately analyzes the data received from the server and prepares it for display as output on the user interface.

[0619] Step 5:

[0620] The device plays received music content and displays navigation instructions. Specifically, while the user explores the city, the device plays music through earphones or speakers, and displays the next destination and route information on the screen. The user can then act accordingly.

[0621] Step 6:

[0622] Users explore and experience the provided content, providing feedback on their devices. Users evaluate the quality of the experience and the accuracy of the guidance, and the device sends this data to the server. This feedback is used to improve the algorithm of the next generation AI model. The output is feedback data.

[0623] (Application Example 1)

[0624] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0625] In modern society, users are expected to efficiently acquire information that matches their preferences and enjoy fulfilling experiences. However, existing information provision systems struggle to provide flexible guidance tailored to individual interests and to deliver personalized, immersive experiences. In particular, in spaces visited by users with diverse interests, there is a need to provide new experiences that integrate music and navigation based on individual preferences.

[0626] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0627] In this invention, the server includes means for acquiring user preference information, means for acquiring user location data, and means for generating music and guidance information based on the preference information and location data. This makes it possible for users to enjoy a personalized musical experience in which they can enjoy music and guidance information tailored to their individual interests.

[0628] "User preference information" refers to information about the interests and concerns of individual users, specifically the distinctive preferences they show regarding particular music genres or places.

[0629] "Location data" refers to geographical information about the user's current location or the location they are planning to move to, and is obtained using GPS or other location measurement technologies.

[0630] "Means for generating music and navigation information" refers to a system that has the function of creating music content and navigation information optimized for the user based on collected user preference information and location data.

[0631] "Means of constructing a musical experience" refers to methods and processes for integrating generated music and informational data to create a new sensory experience for users.

[0632] "Means of delivering a musical experience" refers to the system for actually delivering a constructed musical experience to users, and is usually provided to users through terminals or devices.

[0633] "A spatial route designed based on user interests" refers to a spatial design that takes into account the user's preferences and location data to create a movement path optimized for that individual.

[0634] "Methods for generating custom music" refers to technologies that create music individually tailored to specific situations and interests, based on information obtained from the user's preferences and location.

[0635] In embodiments of the present invention, the entire system consists of the user's smart device and a server. The server acquires location data and preference information transmitted from the user's device and generates music and guidance information using a generative AI model based on this data. The generated content is transmitted to the user's device and provided there as a musical experience.

[0636] The hardware used will be the user's smartphone or mobile device, which will acquire location information. Additionally, the server running the generative AI model will operate on a cloud service. Specific software used will include Python for data analysis and OpenAI, a machine learning library for generating AI models.

[0637] As a concrete example, when a user visits a large shopping center, information on stores that the user is interested in is compiled, and based on that information, a personalized musical guide is generated for the user. This guide combines music that matches the user's interests with store information, supporting a more enjoyable and efficient exploration.

[0638] An example of a prompt message is: "Generate an original musical guide based on the user's interests and design the most appropriate in-store route based on the specified coordinates. Interests: {interest information}, Current location: {location information}". By inputting this prompt message into the AI ​​model, the system provides the user with the most suitable content in real time.

[0639] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0640] Step 1:

[0641] The user's device obtains its current location. It uses GPS to collect the user's location data and sends it to the server. The input is location coordinate data, and the output is location information sent to the server. Specifically, the device's internal location service obtains the location in the background, updates it in real time, and continuously sends it to the server.

[0642] Step 2:

[0643] The server retrieves user preference information. The server obtains individual preference data from the user profile or selected interest categories. The input is user interest information, and the output is an interest dataset. The process involves individually evaluating user preferences based on data periodically sent from the device, and accessing a database on the server to retrieve interest information.

[0644] Step 3:

[0645] The server uses location data and preference information to generate music and navigation information using a generative AI model. The input is location data and preference information, and the output is personalized music content and navigation information. The generative AI model on the server analyzes this input data and generates appropriate content through prompts.

[0646] Step 4:

[0647] The server sends the generated content to the user's terminal. The server transfers data packets containing music and guidance information to the user's device. The output is the data packets received by the terminal. The data transmitted from the server is efficiently packetized according to the network protocol and delivered to the terminal.

[0648] Step 5:

[0649] The system plays music and navigation information received by the user's device as a musical experience. Input is data packets sent from the server, and output is music and navigation received by the user aurally and visually. The device plays the music using an audio player and displays navigation information on the screen.

[0650] Step 6:

[0651] Users enter feedback on the experience provided. This feedback is collected and sent to the server. The input is user feedback, and the output is feedback data used for future generation. Users enter their opinions into forms or questionnaires displayed on their devices, and upon completion, this information is registered in the database.

[0652] Step 7:

[0653] The server utilizes user feedback to improve its generation algorithm. The input is feedback data, and the output is the improved algorithm. By analyzing the feedback information collected on the server and updating the training dataset for the generation AI model, the user can further personalize their experience in subsequent uses.

[0654] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0655] Embodiments of the present invention relate to a system that provides a more personalized entertainment experience by offering a musical experience adapted to the user's emotional state. This system includes terminal, server, and user components, and further incorporates an emotion engine that performs emotion recognition.

[0656] The device acquires emotional data such as the user's facial expressions and voice tone. This data is analyzed in real time within the device or through a connected emotion engine. The results are sent to the server as the user's emotional state. This emotional information, along with other interest information and location information, is also sent to the server.

[0657] The server uses a generative AI model to generate music and guidance text tailored to the user, based on received emotional, interest, and location information. Furthermore, the emotional engine uses data analyzing the user's emotions to dynamically adjust the music's tone and tempo, as well as the content of the guidance text, creating content that best suits the user's current state. For example, if the user is relaxed, it provides calm music and soothing guidance text; if they are energetic, it generates energetic music and dynamic guidance text.

[0658] The server then sends the tailored musical experience content to the device. Through this content, played on the device, users can proactively enjoy a rich experience rooted in their emotions. Users can enjoy music synchronized with their emotions while using their smartphones for everyday commutes and sightseeing, and their interest is further piqued by real-time changing guided text.

[0659] Thus, the present invention opens a new path to making the elimination of sedentary lifestyle enjoyable, while maintaining emotional engagement tailored to the individual user's situation.

[0660] The following describes the processing flow.

[0661] Step 1:

[0662] The device, with the user's permission, uses the smartphone's camera and microphone to capture the user's facial expressions and voice, thereby acquiring emotional data. The emotion engine analyzes this data in real time to identify the user's emotional state.

[0663] Step 2:

[0664] The device transmits user interest information and location information to the server along with acquired emotional state data. This information serves as foundational data for building a personalized user experience.

[0665] Step 3:

[0666] The server comprehensively analyzes received emotional, interest, and location information. It then uses an AI model to generate music content and guidance text best suited to the user's current situation. During this process, the atmosphere of the music and the tone of the guidance text are dynamically adjusted according to the user's emotional state.

[0667] Step 4:

[0668] The server sends the generated, personalized musical experience content to the device. This content is optimized based on the user's past history and current emotions.

[0669] Step 5:

[0670] The device plays content received from the server, providing users with a musical experience. Users can explore the city while on the go through music and guidance synchronized with their emotions, and enjoy real-time updated information.

[0671] Step 6:

[0672] Users can provide feedback based on their experience through their device. The device sends this feedback to the server. The server analyzes the collected feedback and uses it to improve algorithms that further enhance the user experience in future content creation.

[0673] (Example 2)

[0674] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0675] Modern entertainment systems face challenges in providing individually customized content based on a user's emotional state, potentially leading to decreased user satisfaction. Furthermore, there is a lack of systems that effectively utilize user interest and location information to deliver optimal entertainment in real time. This invention aims to improve the personalized entertainment experience by considering the user's emotional state and generating more individualized voice and guidance text.

[0676] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0677] In this invention, the server includes means for acquiring user emotional data, means for analyzing the acquired emotional data, and means for acquiring user interest information. This makes it possible to personalize voice and guidance text based on the user's emotional state and provide a user-optimized entertainment experience in real time.

[0678] "Emotional data" refers to information that indicates a user's emotional state, obtained from things like their facial expressions and tone of voice.

[0679] "Analysis" refers to the process of processing acquired emotional data to identify or classify the user's emotions.

[0680] "Interest information" refers to information about a user's interests and preferences, and is data used to generate personalized entertainment experiences.

[0681] "Location information" is data that indicates the user's current geographical location and is used to provide content based on the user's movement and surrounding environment.

[0682] "Sound" refers to music and sounds generated based on the user's emotional state, and is an element that enriches the user's experience.

[0683] An "informative text" is a text or audio message associated with audio that provides information to the user.

[0684] An "entertainment experience" is a series of user experiences that combine customized audio and guidance text generated based on emotional data, interest information, and location information.

[0685] "Real-time" refers to a process where information processing and content generation are performed in response to the user's current state.

[0686] This invention is a system that provides personalized entertainment experiences based on the user's emotional state. The system consists of a terminal, a server, and a user, and utilizes the user's emotions, interests, and location information to generate optimal voice and guidance text.

[0687] The terminal is a device equipped with a camera and microphone as a means of acquiring user emotional data. Specifically, it can utilize common electronic devices such as smartphones and tablets. The terminal uses the sensors of these devices to collect the user's facial expressions and voice tone in real time and analyzes them using an emotion engine. This analysis makes it possible to understand the user's emotional state.

[0688] The server receives emotional information, interest information, and location information transmitted from the terminal, and combines this data to generate voice and guidance text using a generative AI model. The generative AI model can utilize, for example, open-source natural language processing models or commercial AI services. This makes it possible to adjust the voice tone and guidance text content according to the user's emotions.

[0689] Users utilize this system while carrying their devices and moving around. For example, by going out with their smartphone and walking around, they can receive music that matches their mood and real-time guidance. This allows users to enjoy a richer experience. An example of a specific prompt message is sending the instruction, "Generate calming music suitable for when the user is relaxed," to the generation AI model. Based on such prompt messages, the system can provide optimal content that meets the user's needs.

[0690] Thus, this system is designed to provide a dynamic and personalized entertainment experience that takes into account the user's emotions and activities.

[0691] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0692] Step 1:

[0693] The device uses a camera and microphone to acquire emotional data such as the user's facial expressions and voice tone. The input is the user's physical facial expressions and voice, while the output is digitized emotional data. This data is collected in real time for use in subsequent analysis steps.

[0694] Step 2:

[0695] The device analyzes the acquired emotional data using an emotion engine. This analysis uses a machine learning model to evaluate facial feature points and voice frequency components to identify the emotional state. The input is the emotional data obtained in step 1, and the output is an emotion label such as "happy," "sad," or "surprised." This quantifies the user's emotional state.

[0696] Step 3:

[0697] The device collects user interest information and location information along with emotional information, and sends them to the server. The input for this step is the analyzed emotional information, pre-configured interest information, and location information obtained via GPS, etc., and the output is the data to be sent to the server. The data is transmitted securely through a secure communication method.

[0698] Step 4:

[0699] The server uses the received emotion, interest, and location information to send prompt messages to the generating AI model. This input is the combined information obtained in step 3, and the model generates voice and guidance text. The output is an audio file and guidance text optimized for the user's situation. This process creates personalized content based on the user's emotions.

[0700] Step 5:

[0701] The server sends the generated audio and guidance text to the terminal. The input for this step is the generated content obtained in step 4, and the output is its transfer to the terminal. The terminal receives this transmitted data and begins playing it at a time appropriate for the user.

[0702] Step 6:

[0703] The terminal plays the transmitted audio and guidance text, providing the user with an entertainment experience. The input is the audio file and guidance text received in step 5, and the output is the audiovisual content that the user actually experiences. In this step, the playback content may be dynamically adjusted in response to the user's reactions and the environment while moving.

[0704] (Application Example 2)

[0705] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0706] Traditional entertainment experiences have struggled to provide content tailored to individual user emotions. As a result, users are often provided with experiences that are not necessarily optimal, failing to fully realize the appeal of entertainment. In particular, virtual environments often lack a sense of presence and immersion, making it essential to improve user engagement by providing personalized experiences.

[0707] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0708] In this invention, the server includes means for acquiring user interest information, means for acquiring user location information, means for analyzing user emotional information, and means for generating music and guidance text based on the interest information, location information, and emotional information. This enables a personalized musical experience synchronized with the user's emotions, enhancing realism and immersion, and providing a more individualized and optimized entertainment experience.

[0709] "User interest information" refers to data about things and themes that users are currently interested in, and is used to generate music and explanatory text.

[0710] "User location information" refers to data about the geographical location where the user is currently located, and is used to provide information about the surrounding area and appropriate content.

[0711] "User emotional information" refers to data that indicates the user's emotional state, analyzed from the user's facial expressions and tone of voice.

[0712] "Means for generating music and guidance text" refers to a process or apparatus for creating individually tailored music and guidance text based on the user's interest information, location information, and emotional information.

[0713] A "musical experience synchronized with the user's emotions" is an immersive entertainment experience created by combining music and explanatory text that are adjusted to the user's emotional state.

[0714] The system for realizing this invention consists of a user terminal, a server, and an emotion engine. The terminal is envisioned as smart glasses or a head-mounted display and is equipped with a camera and microphone to capture the user's facial expressions and voice tone. In particular, this emotion information is analyzed in real time and sent to the emotion engine.

[0715] The server is either run on specific hardware or located in the cloud. It houses modules for collecting user interest information, acquiring location information, and software necessary for analyzing sentiment information. Specific software includes OpenCV for image recognition and Google Cloud Speech-to-Text for speech analysis.

[0716] Using a generative AI model, the server generates music and explanatory text. Examples of generative AI models include natural language generation engines such as GPT-3 and Bard. This allows the music's tone, tempo, and the content of the explanatory text to be adjusted according to the user's emotions. The generated content is then sent to the user's device.

[0717] As a concrete example of creating a musical experience that matches the user's emotions, consider a scenario where a user is using smart glasses while sightseeing. If the user's facial expression indicates enjoyment, the system plays calm and cheerful music and provides guidance such as, "There is a beautiful view nearby."

[0718] An example of a prompt from this system would be, "If the user's emotions are perceived as calm, generate calming guidance text about soothing music and scenery." This prompt allows the server to utilize its AI generation model to provide the user with the most optimal experience.

[0719] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0720] Step 1:

[0721] The device acquires the user's facial expressions and voice. Input consists of image data captured by the camera and voice data collected by the microphone. This data is sent to the emotion engine, which is then prepared for emotion analysis.

[0722] Step 2:

[0723] The emotion engine analyzes received facial expression data and voice data. It uses OpenCV for image processing and Google Cloud Speech-to-Text for voice analysis. The input is the user's facial expressions and voice tone, and the output is data indicating the user's emotional state. This analysis quantifies the emotions the user is currently experiencing.

[0724] Step 3:

[0725] The server collects user interest and location information. Inputs include user profile data and location information obtained from GPS sensors, while outputs are the information necessary for generating content based on interests and location.

[0726] Step 4:

[0727] The server generates music and guidance text using a generative AI model. The inputs here are emotional information, interest information, and location information. Based on this, prompt text is created and fed to the generative AI model (e.g., GPT-3) to output music and guidance text synchronized with the user's emotions.

[0728] Step 5:

[0729] The server combines the generated music and explanatory text to create a musical experience. The output is new content, including music and explanatory text that matches the emotional state, and is sent to the terminal.

[0730] Step 6:

[0731] The device provides the user with received musical experience content. The input is content sent from the server, and the output is what the user sees. The user enjoys a personalized experience combining music and explanatory text through smart glasses.

[0732] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0733] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0734] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0735] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0736] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0737] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0738] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0739] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0740] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0741] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0742] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0743] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0744] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0745] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0746] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0747] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0748] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0749] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0750] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0751] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0752] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0753] The following is further disclosed regarding the embodiments described above.

[0754] (Claim 1)

[0755] Means of obtaining user interest information,

[0756] Means for obtaining the user's location information,

[0757] A means for generating music and guidance text based on the aforementioned interest information and location information,

[0758] A means of generating a musical experience by combining the aforementioned music and explanatory text,

[0759] The means of providing the aforementioned musical experience,

[0760] A system that includes this.

[0761] (Claim 2)

[0762] The system according to claim 1, further comprising means for providing the user with information about nearby spots as navigation.

[0763] (Claim 3)

[0764] The system according to claim 1, further comprising means for collecting user feedback and improving the generation algorithm.

[0765] "Example 1"

[0766] (Claim 1)

[0767] Means of obtaining user interest information,

[0768] Means for obtaining the user's location information,

[0769] A means for generating prompt data using generative artificial intelligence based on the aforementioned interest information and location information, and for generating music and guidance text,

[0770] A means of generating a musical experience through a user interface by combining the aforementioned music and explanatory text,

[0771] A means of providing the aforementioned musical experience, receiving user feedback, and incorporating it into the next generation,

[0772] A system that includes this.

[0773] (Claim 2)

[0774] The system according to claim 1, further comprising means for providing the user with surrounding geographical information as navigation and outputting music content using audio equipment.

[0775] (Claim 3)

[0776] The system according to claim 1, further comprising means for recording user feedback and improving the generation algorithm of the generating artificial intelligence.

[0777] "Application Example 1"

[0778] (Claim 1)

[0779] Means for obtaining user preference information,

[0780] A means of obtaining user location data,

[0781] A means for generating music and guidance information based on the aforementioned preference information and location data,

[0782] A means of integrating the aforementioned music and informational data to construct a musical experience,

[0783] The means of providing the aforementioned musical experience,

[0784] A means of generating spatial paths and custom music designed based on user interests,

[0785] A means of combining them to provide the optimal store route,

[0786] A system that includes this.

[0787] (Claim 2)

[0788] The system according to claim 1, further comprising means for presenting design information of the surrounding area to the user as navigation.

[0789] (Claim 3)

[0790] The system according to claim 1, further comprising means for collecting user feedback and improving the generation mechanism.

[0791] "Example 2 of combining an emotion engine"

[0792] (Claim 1)

[0793] Means for acquiring user sentiment data,

[0794] A means of analyzing acquired emotional data,

[0795] Means of obtaining user interest information,

[0796] Means for obtaining the user's location information,

[0797] Means for generating voice and guidance text based on the aforementioned emotional information and interest information and location information,

[0798] A means for generating an entertainment experience by combining the aforementioned audio and guidance text,

[0799] Means for providing the aforementioned entertainment experience,

[0800] A system that includes this.

[0801] (Claim 2)

[0802] The system according to claim 1, further comprising means for providing the user with information about nearby locations as route guidance.

[0803] (Claim 3)

[0804] The system according to claim 1, further comprising means for collecting user responses and improving the generation algorithm.

[0805] "Application example 2 when combining with an emotional engine"

[0806] (Claim 1)

[0807] Means of obtaining user interest information,

[0808] Means for obtaining the user's location information,

[0809] A means of analyzing user sentiment information,

[0810] A means for generating music and guidance text based on the aforementioned interest information, location information, and emotional information,

[0811] A means for generating a musical experience synchronized with the user's emotions by combining the aforementioned music and explanatory text,

[0812] A means of providing the aforementioned musical experience using a visual display device,

[0813] A system that includes this.

[0814] (Claim 2)

[0815] The system according to claim 1, further comprising means for providing information on nearby tourist attractions as navigation and adjusting it according to the user's emotional state.

[0816] (Claim 3)

[0817] The system according to claim 1, further comprising means for collecting user feedback and sentiment data and improving generation algorithms to provide more personalized content. [Explanation of symbols]

[0818] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. Means of obtaining user interest information, Means for obtaining the user's location information, A means for generating music and guidance text based on the aforementioned interest information and location information, A means of generating a musical experience by combining the aforementioned music and explanatory text, The means of providing the aforementioned musical experience, A system that includes this.

2. The system according to claim 1, further comprising means for providing the user with information about nearby spots as navigation.

3. The system according to claim 1, further comprising means for collecting user feedback and improving the generation algorithm.

Images