system

Abstract

We provide the system. [Solution] A means of detecting a user's free time, A means of extracting user usage history and interests, A means of proposing activities that will satisfy users, A means of notifying users of the proposal, A means of collecting user feedback and updating the proposed model, A means for presenting real-time activity information based on spatial circumstances via a visual information display device, A system that includes this.

Description

Technical Field

[0004] , , ,

[0005] , , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance that responds to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern busy lives, especially among the younger generation, efficient utilization of time is required. However, it is not easy to find a way to make the most of limited free time and enhance individual satisfaction. When users find appropriate activities from many options, decision - making may become difficult due to information overload or forgetfulness. Therefore, there is a need for a system that proposes appropriate activities based on the user's interests and past behaviors.

Means for Solving the Problems

[0005] This invention provides a system that detects a user's free time, extracts their usage history and interests, and then suggests activities that are likely to satisfy the user. Furthermore, the system notifies the user of the suggestions, records the user's response, and updates the suggestion model, thereby improving the accuracy of future suggestions. In addition, by combining means of estimating the user's interests by analyzing data related to the user's past electronic information exchanges and means of optimizing suggestions using the user's current location information, a more personalized experience is achieved.

[0006] "Methods for detecting a user's free time" refer to functions that analyze a user's schedule from calendars and scheduling apps to identify time slots that are freely available.

[0007] "Means for extracting user usage history and interests" refers to algorithms or functions that analyze a user's past actions and activities of interest, and use that information to identify the user's preferences.

[0008] "A means of suggesting highly satisfying activities to users" refers to a function that selects activities deemed to be of high value based on the user's free time and interests, and then informs the user of those activities.

[0009] "Means of notifying users of suggestions" refers to a function that sends selected activities as notifications to the user's device, allowing the user to receive that information.

[0010] "Means for recording user responses and updating the suggestion model" refers to a function that collects how users responded to suggestions and uses that information to modify the algorithm in order to improve the accuracy of future suggestions.

[0011] "A means of analyzing data related to a user's past electronic information exchanges to estimate the user's interests" refers to a function that analyzes usage history of social media, email, etc., to predict what topics and activities the user is interested in.

[0012] "A means of optimizing suggestions by utilizing the user's current location information" refers to a function that identifies the user's current geographical location and selects and suggests activities that can be performed in the vicinity of that location. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] The system of this invention is designed to enable users to efficiently utilize their time in their daily lives and engage in highly satisfying activities. This system improves the user's quality of life by detecting their free time and suggesting activities that match that time.

[0035] The server plays a central role, first obtaining the user's schedule information. This identifies the user's free time. Next, the server analyzes the user's social media data and other usage history to extract the user's interests and past activities. Based on this information, it selects activities that are likely to be highly satisfying for the user within the given time constraints.

[0036] The device's role is to notify the user of recommended activities received from the server. Notifications are customized according to the user's preferences, allowing for options such as push notifications or email. The user can review the notification and either add the suggested activity to their schedule or decline the suggestion.

[0037] When a user accepts or rejects an activity, the device sends this response as feedback to the server. The server uses this feedback to adjust its suggestion algorithm and improve the accuracy of future suggestions.

[0038] As a concrete example, consider a scenario where a user has an hour of free time during their lunch break. The server detects this free time and suggests an activity: visiting a nearby gallery that the user has previously expressed interest in on social media. The device notifies the user of this suggestion, and if the user accepts, the activity is automatically added to their schedule.

[0039] In this way, the present invention efficiently proposes activities that match the user's interests and needs within time constraints, thereby supporting a fulfilling lifestyle for the user.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The server accesses the user's schedule database to retrieve the user's appointments. This allows it to identify available time slots and generate free time data.

[0043] Step 2:

[0044] The server collects data related to the user's past posts and interests through SNS APIs. Furthermore, it uses machine learning models to analyze activities and themes that the user is likely to be interested in.

[0045] Step 3:

[0046] The server combines free time data with user interest data to select the most suitable activity. The selection is optimized to suggest beneficial activities within the user's accessible range, taking into account their current location.

[0047] Step 4:

[0048] The device receives suggested activities from the server and displays a notification to the user. The notification format is determined according to the user's settings, and may include push notifications or email.

[0049] Step 5:

[0050] The user reviews the device notification and chooses to accept or reject the suggested activity. The user's choice is recorded by the device.

[0051] Step 6:

[0052] The terminal sends the user's selection results to the server. The server analyzes the results and adjusts the suggestion algorithm. By adjusting the algorithm, the accuracy of future suggestions is improved, and the user's preferences are reflected more accurately.

[0053] (Example 1)

[0054] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0055] In today's information-driven society, individuals have limited time to dedicate to enriching activities based on their hobbies and interests amidst their busy daily lives. Therefore, there is a need for a system that efficiently suggests activities suited to individual free time, thereby improving users' quality of life. However, conventional systems often fail to adequately reflect user interests in their suggestions, and improving the accuracy of real-time suggestions is challenging.

[0056] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0057] In this invention, the server includes means for acquiring the user's time information and identifying spatial availability, means for analyzing the user's interests and past behavioral history, and means for selecting activities that will provide high satisfaction to the user using a generating AI. This makes it possible to suggest activities that are customized to each individual user and are highly accurate and delivered in real time.

[0058] "User time information" refers to information about an individual's available time within a specific period, such as their schedule or appointments.

[0059] "Spatial leeway" refers to unscheduled free time within a user's schedule, providing a gap in time for engaging in new activities.

[0060] "Interest" refers to the matters that a user shows interest in or attention to regarding a particular theme or field.

[0061] "Past behavioral history" refers to a record of various activities and choices a user has made in the past, and serves as data for analyzing their interests and tendencies.

[0062] "Generative AI" refers to a model or method that uses artificial intelligence technology to automatically generate new data, information, or suggestions.

[0063] A "high-satisfaction activity" is an activity that users find interesting and that is considered to improve their quality of life.

[0064] "Information and communication technology" is a general term for technologies and methods for efficiently transmitting, receiving, and processing information.

[0065] "Digital data" refers to information in a format handled by computers and digital devices, which is generated, stored, and transmitted electronically.

[0066] An "activity suggestion algorithm" is a set of calculation procedures and rules used to select activities that are suitable for the user.

[0067] "Location information" is data that indicates a user's geographical location at a specific time, and is used to optimize activities.

[0068] In order to implement this invention, the server, terminal, and user each need to play a specific role.

[0069] The server uses an API from a cloud-based scheduling tool to retrieve the user's schedule information. This tool retrieves the user's schedule information as digital data and stores it in JSON format. Specifically, it is expected that scheduling tools such as Google® Calendar API will be used. The server processes this information to identify the user's time information.

[0070] Next, the server uses social media APIs to analyze the user's interests and past behavior history. For example, it retrieves user "likes" and follow information through the Twitter and Instagram APIs and analyzes it using the Python pandas library. This process identifies the user's areas of interest, and the results are used by the generative AI to select activities that will provide high satisfaction. The generative AI model used is one designed to generate activities based on the user's behavior.

[0071] An example of a prompt for the generating AI model is: "Consider the user's schedule and social media data, and suggest the most suitable activity for their next free time. Example: 1-hour lunch break, interest: art."

[0072] The device serves to notify the user of suggested activities. It uses push notification services such as Firebase Cloud Messaging to communicate selected activities to the user using information and communication technology. These notifications can be customized to the user's preferences, and email notifications are also an option.

[0073] Users can check notifications from their devices and choose to accept or reject suggested activities. This response is sent back to the server and recorded as digital data. Using this data, the server can adaptively improve the activity suggestion algorithm and increase the accuracy of future suggestions.

[0074] In this way, users, servers, and terminals work together to dynamically suggest activities that are optimal for each individual user, thereby improving the user's quality of life.

[0075] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0076] Step 1:

[0077] The server retrieves the user's schedule information. The input is the user's schedule data obtained from the API of a cloud-based scheduling tool. The server analyzes this data, uses a Python library to identify spatial availability, and identifies the next available time slots. The output is a list of available time slots. Specifically, the server requests schedule information using the API key and analyzes the returned data.

[0078] Step 2:

[0079] The server collects and analyzes user interests and past behavioral history. The input is user "likes" and follow information obtained through SNS APIs. The server uses the pandas library to organize the data and extract specific areas of interest. The output is a list of keywords indicating the user's interests. Specifically, the server sends API requests to SNS accounts to collect interest data.

[0080] Step 3:

[0081] The server uses a generative AI model to select activities that will provide high satisfaction to the user. The input consists of a list of free time slots and a list of keywords of the user's interests. The server inputs these as prompts into the generative AI model to generate the optimal activity. The output is the recommended activity. Specifically, the server inputs prompts into the generative AI and analyzes the results obtained.

[0082] Step 4:

[0083] The device notifies the user of recommended activities received from the server. The input is the recommended activities sent from the server. The device uses this information to send a push notification to the user. The output is the notification displayed on the user's device. Specifically, the device uses Firebase Cloud Messaging to send the notification.

[0084] Step 5:

[0085] The user checks the notification from their device and accepts or rejects the offered activity. The input is the notification from the device. The user's choice is recorded as an action and returned to the server as feedback. The output is the result of the user's choice. In concrete terms, the user inputs their choice through the interface on their device.

[0086] Step 6:

[0087] The server receives user selections as feedback and updates the activity suggestion algorithm. The input is the user's selection results. The server analyzes this data and adjusts the algorithm's weights. The resulting output is an updated algorithm designed to improve the accuracy of future suggestions. Specifically, the server saves the feedback to a database and retrains the algorithm.

[0088] (Application Example 1)

[0089] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0090] In modern times, many people find it difficult to manage their time in their daily lives, and there is a particular need to make efficient use of their free time. However, there are limited systems that can suggest meaningful activities tailored to individual lifestyles and interests, and these have not been able to increase user satisfaction. Furthermore, there is a need for suggestions using visual means that can provide more intuitive and immediate information.

[0091] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0092] In this invention, the server includes means for detecting the user's free time, means for extracting the user's usage history and interests, and means for presenting real-time activity information based on spatial circumstances via a visual information display device. This enables efficient and satisfying use of time by suggesting optimized activities to the user and notifying them of these suggestions visually and intuitively.

[0093] "Means for detecting a user's free time" refers to a function that analyzes a user's schedule information to identify time periods when normal activities are not scheduled.

[0094] "Means for extracting user usage history and interests" refers to functions that analyze past behavioral data and records of electronic information exchange to understand the topics of interest and behavioral trends of users.

[0095] "A means of suggesting highly satisfying activities to users" refers to a function that selects the most suitable activities based on the user's free time and interests, and makes recommendations to increase satisfaction.

[0096] "Means of notifying users of suggestions" refers to a means of informing users of selected activities, which is a function that displays or notifies users of information on their devices using visual or auditory methods.

[0097] "Means for collecting user responses and updating the suggestion model" refers to a function that accumulates user responses to suggestions and continuously improves the activity suggestion algorithm based on that data.

[0098] "Means for presenting real-time activity information based on spatial circumstances via a visual information display device" refers to a function that visually displays real-time location information and activities corresponding to the surrounding environment to the user, thereby concretizing the information.

[0099] This system is designed to help users make effective use of their free time and support a more fulfilling lifestyle. The server retrieves the user's schedule information and identifies their free time. This process involves analyzing schedule information stored in a cloud-based database management system. Next, the server uses machine learning models to analyze SNS data and past electronic information exchange data to extract the user's usage history and interests.

[0100] The device is a visual information display device, such as smart glasses, that notifies the user of optimal activity suggestions received from a server. These suggestions include real-time recommendations that take spatial context into account, based on the user's current location. For example, while a user is walking in the city, information about events taking place in a nearby park might be displayed. This intuitive display allows the user to consciously choose their activity.

[0101] Furthermore, user feedback is constantly fed back from the device to the server, contributing to the improvement of the proposed algorithm's accuracy. This feedback is used to train the generative AI model, which then adjusts subsequent proposals to better match user needs.

[0102] For example, if a user is at a cafe during their lunch break on a weekday, the server will recommend beneficial activities in the vicinity based on their location. An example of a prompt to input into the generating AI model would be, "Based on the user's schedule and social media data, please suggest activities that are relevant to their current location." Through this entire process, it is possible to enrich the user's life.

[0103] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0104] Step 1:

[0105] The server retrieves user schedule data from a cloud database. The input is the user's schedule information, which is used to identify free time. Specifically, it uses a database management system to access and analyze the user's calendar data to detect periods of free time.

[0106] Step 2:

[0107] The server acquires SNS data and past electronic information history, and uses a generative AI model to extract user interests. The input is SNS data and usage history, and the output is the user's areas of interest. Specifically, it analyzes text data using natural language processing techniques and lists the user's areas of interest.

[0108] Step 3:

[0109] Based on the above information, the server selects suitable activities and generates activity information optimized in real time based on the user's current location. Inputs include free time, interests, and location information, while output is activity suggestions. The server uses location services to search, select, and recommend events and activities relevant to the user's location.

[0110] Step 4:

[0111] The smart glasses, acting as the terminal, receive suggestions from the server and present them to the user as visual notifications. The input is activity suggestions from the server, and the output is a visual display. Specific actions include displaying information on the screen in a conspicuous manner through the user interface.

[0112] Step 5:

[0113] The user reviews the suggested activity and responds by selecting or rejecting it. The input is a visual notification, and the output is the user's choice. The user looks at the displayed suggestions, makes a selection based on their interests, and sends feedback to the server via the terminal.

[0114] Step 6:

[0115] The server receives user feedback and updates the proposed algorithm. The input is the user's selection and feedback, and the output is the updated proposed model. Specifically, the feedback is input into the machine learning algorithm to tune the model and improve the accuracy of future proposals.

[0116] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0117] The present invention is an activity suggestion system that incorporates an emotion engine to efficiently utilize users' free time and improve user satisfaction. This system recognizes the user's emotional state in real time and adjusts the suggested activities based on that information.

[0118] The server retrieves the user's schedule information and identifies their free time. Next, the server analyzes the user's social media data and past usage history to extract their interests. During this process, the emotion engine analyzes the user's emotional state. Emotions are evaluated in real time using text analysis and image recognition technologies. Based on these evaluation results, the server selects activities to suggest.

[0119] As a concrete example, consider a scenario where a user is feeling stressed and needs to be offered relaxing activities. The server detects signs of stress from the user's facial expressions and text information, and then suggests the most suitable relaxation spot based on the user's time and interests. For example, it might recommend a walk in a nearby park or reading in a quiet cafe.

[0120] The device receives suggestions from the server and notifies the user. Notifications are tailored to the user's settings, with priority given to activities such as stress relief. The user checks the device's notifications and accepts the suggestions, which are automatically added to the schedule.

[0121] Furthermore, by recording user responses, the server can update its suggestion algorithm, enabling it to propose more effective activities in the future. This system leverages emotion engine data to provide a dynamic and personalized experience tailored to the user's psychological state.

[0122] The following describes the processing flow.

[0123] Step 1:

[0124] The server retrieves user schedule data from the database. This identifies the user's current appointments and free time, and prepares the data for analysis.

[0125] Step 2:

[0126] The server collects users' past posts and activity history through APIs from social media and other platforms. The collected data is then analyzed using text to extract user interests and preferences.

[0127] Step 3:

[0128] The emotion engine identifies emotions from collected content to determine the user's situation. It uses keywords in text, as well as data obtained from audio and images, to assess the current emotional state.

[0129] Step 4:

[0130] Based on the interests and emotions identified by the server, the most suitable activities are selected. For example, for a user experiencing stress, activities that contribute to relaxation are prioritized.

[0131] Step 5:

[0132] The device receives recommended activities from the server and displays them to the user as notifications. These notifications are presented prominently within the user interface and designed to attract the user's attention.

[0133] Step 6:

[0134] Users review notifications and accept or reject suggested activities. If accepted, the activity is automatically added to the user's schedule, reducing the effort required for discovery.

[0135] Step 7:

[0136] The device sends the user's selection as feedback to the server. This feedback is used to refine the algorithm and improve the accuracy of future suggestions.

[0137] This process enables personalized activity suggestions that are tailored to the user's emotions and preferences.

[0138] (Example 2)

[0139] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0140] In modern society, many people are unaware of their own emotional state due to the busyness and stress of daily life, and as a result, are unable to effectively utilize their free time. There is a need to improve user satisfaction and reduce stress by suggesting optimal activities based on the user's emotional state and interests.

[0141] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0142] In this invention, the server includes means for detecting the user's free time, means for analyzing the user's emotional state in real time, and means for extracting the user's interests based on the information. This makes it possible for the user to make the most of their free time in the most optimal way.

[0143] "Free time" refers to periods in a user's schedule when no specific appointments are scheduled.

[0144] "Emotional state" refers to information that indicates the user's current psychological or emotional condition, and is derived from text and images.

[0145] "Real-time analysis" refers to processing user data instantly without any time delay and obtaining results immediately.

[0146] "Interest extraction" refers to identifying themes and activities that a user is interested in, based on their past behavior and digital information.

[0147] "Activity suggestions" refer to recommending specific actions or activities to users based on their emotional state and interests.

[0148] "Notifications" refer to sending messages or alerts to inform users about suggested activities.

[0149] "Response recording" refers to saving data on the actions and choices a user makes in response to suggested activities.

[0150] "Updating the suggestion algorithm" means improving the algorithm based on recorded user feedback to enhance the accuracy of future suggestions.

[0151] The present invention is a system that uses multiple computer resources to provide activity suggestions that take into account the user's emotional state. The server first obtains the user's schedule information using a specific API and identifies available time. In this process, data is collected, for example, from a calendar application.

[0152] Next, the server uses natural language processing tools (e.g., spaCy and NLTK) to analyze the user's social media data and past usage history, extracting the user's interests. It also utilizes the Sentiment Analysis API and image recognition technologies (e.g., OpenCV and Amazon Rekognition) as sentiment engines to evaluate the user's emotional state in real time from their text information and photos. This evaluation provides the foundational data needed to guide appropriate activities.

[0153] The server utilizes the user's current location information and leverages geographic information services (e.g., Google Maps API) to optimize activity suggestions. This allows it to select activities available near the user and suggest relaxing spots and entertainment options tailored to the user's emotional state.

[0154] The device is responsible for notifying the user of suggestions from the server. Notifications are delivered via push notifications or email according to the user's preferences, and features such as prioritizing activities that take emotional state into consideration are implemented. When the user accepts a suggestion, it is automatically added to the schedule.

[0155] For example, if the server determines from a user's social media posts or images that they are "feeling stressed," it can suggest a walk in a nearby park or a quiet cafe. The AI ​​model generates prompts like the following to suggest activities that can help the user relax: "Please suggest activities that can help the user relax when they are feeling stressed."

[0156] User responses are recorded, and this data is analyzed by the server and used to update the proposed algorithm. This feedback allows the system to provide optimal, personalized activities for the user over the long term.

[0157] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0158] Step 1:

[0159] The server retrieves the user's schedule information. The input is data from the user's calendar application. The server analyzes this information to identify free time slots that are not scheduled. The output is a list of the user's next free time slots. Specifically, the server retrieves data from the calendar application via an API and determines whether there are appointments or not for each date and time.

[0160] Step 2:

[0161] The server analyzes the user's social media data and past usage history to extract their interests. The input consists of the user's past web browsing history and social media posts. The server processes this data using natural language processing tools to extract specific keywords and topics. The output is a list of keywords indicating the user's interests. Specifically, the server uses a text analysis engine to evaluate frequently occurring words and contexts to identify areas of interest.

[0162] Step 3:

[0163] The server evaluates the user's real-time emotional state. The input is the text and images from the user's latest social media posts. The server uses an emotion engine to analyze the sentiment of the text and images and determine the user's current emotional state. The output is a positive, negative, or neutral emotional state. Specifically, the server uses an emotion analysis API to aggregate positive and negative scores for words, and also uses image recognition to infer emotions from facial expressions.

[0164] Step 4:

[0165] The server selects appropriate activities based on the user's free time, interests, and emotional state. The inputs are free time, user interests, and emotional state. The server uses a generative AI model to generate the optimal activity from this data. The output is the activity suggested to the user. Specifically, the server dynamically generates different activity candidates, evaluates them, and selects the most suitable one.

[0166] Step 5:

[0167] The device notifies the user of selected activity suggestions. The input is the activity suggestion sent from the server. The device sends a push notification to deliver this information to the user. The output is the activity suggestion message displayed on the user's device. Specifically, the device activates the notification system and delivers a message to the user accompanied by an alarm or vibration depending on the importance level.

[0168] Step 6:

[0169] The user reacts to the suggested activity, and the result is recorded. The input is the user's reaction, i.e., their acceptance or rejection of the suggestion. The server stores this information in a reaction log and uses it to improve the suggestion algorithm. The output is the updated algorithm. Specifically, the server periodically takes the user's selection history as training data and forms a feedback loop to improve the accuracy of suggestions in the future.

[0170] (Application Example 2)

[0171] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0172] In modern society, many users experience stress and fatigue in their daily lives. Furthermore, it is difficult to suggest optimal activities based on users' emotional states and interests amidst their busy schedules. Therefore, this invention aims to improve user satisfaction by efficiently utilizing users' free time and suggesting activities that match their emotional state.

[0173] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0174] In this invention, the server includes means for detecting the user's free time, means for extracting the user's usage history and interests, and means for analyzing the user's emotional state using an emotion recognition engine and adjusting the suggested activities. This makes it possible to suggest appropriate activities to the user in real time according to their emotional state.

[0175] "User free time" refers to the period of time when a user has no scheduled appointments and is free to use as they please.

[0176] "Usage history" refers to data that records information about activities and services a user has used in the past.

[0177] "Interest" refers to information related to the subjects or actions that users are interested in.

[0178] An "emotion recognition engine" refers to a program or system that analyzes a user's emotional state from their facial expressions, voice, and text data.

[0179] "Real-time evaluation" refers to a process that analyzes and outputs results instantly at the moment data is generated.

[0180] "Activities" refer to actions, tasks, or activities that become part of a user's life, such as relaxation.

[0181] "Notifications" refer to sending messages or alerts to inform users of information or suggestions.

[0182] "Response" refers to the user's reply or action in response to a suggestion or notification.

[0183] A "proposal model" refers to an algorithm or mechanism for selecting content to propose to the user.

[0184] In this embodiment of the invention, the system primarily consists of a server and terminals. The server has the capability to detect users' free time and process large amounts of data to extract usage history and interests. This includes cloud-based data storage and an analysis engine. The server also analyzes the user's real-time emotional state using an emotion recognition engine. To this end, it uses OpenCV for image recognition technology and a virtual emotion recognition library for emotion analysis.

[0185] The device functions as a user interface, notifying the user of activity suggestions. These notifications can be customized according to user settings; for example, suggestions aimed at stress relief can be prioritized. The device functions as a smartphone or consumer robot, recording user responses and sending feedback to a server.

[0186] This feedback loop allows the suggestion model to be continuously updated and evolve to recommend even more personalized activities to the user. For example, if the device determines that a user is stressed upon returning home, it may play soothing music to create a relaxing environment.

[0187] An example of a prompt suitable for a generative AI model is: "Analyze the user's facial expressions from the image captured by the camera, and if stress is detected, suggest a relaxing activity."

[0188] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0189] Step 1:

[0190] The server retrieves user schedule information and identifies available time slots. The input is the user's schedule data, and the output is the identified available time slots. This process fetches schedules from a database and extracts available time slots using an algorithm.

[0191] Step 2:

[0192] The server analyzes the user's social media data and past usage history to extract their interests. Input is social media posts and past activity logs, and output is information about the user's interests. Data analysis tools are used, and text mining techniques are employed to identify areas of interest.

[0193] Step 3:

[0194] The server uses an emotion recognition engine to analyze the user's emotional state in real time. Input is the user's facial image and voice data, and output is the analyzed emotional state. It utilizes OpenCV and a virtual emotion recognition library to scan facial expressions and voice patterns.

[0195] Step 4:

[0196] The server selects appropriate activities based on the user's free time, interests, and emotional state. The input is free time, interests, and emotional information, and the output is suggested activities. An algorithm is used to generate recommended activities by considering multiple factors.

[0197] Step 5:

[0198] The device receives suggestions from the server and notifies the user. The input is suggestion activity information, and the output is the notification to the user. The notification is displayed on the device as a pop-up or voice guidance.

[0199] Step 6:

[0200] The user checks the notification on their device and selects or rejects an activity based on it. The input is the suggested activity, and the output is the user's response. By selecting an option, the user determines their next action.

[0201] Step 7:

[0202] The server records user responses and updates the suggestion model. The input is user response data, and the output is the updated model. This allows future suggestions to be more personalized and the model to better suit the user's preferences.

[0203] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0204] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0205] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0206] [Second Embodiment]

[0207] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0208] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0209] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0210] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0211] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0212] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0213] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0214] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0215] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0216] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0217] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0218] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0219] The system of this invention is designed to enable users to efficiently utilize their time in their daily lives and engage in highly satisfying activities. This system improves the user's quality of life by detecting their free time and suggesting activities that match that time.

[0220] The server plays a central role, first obtaining the user's schedule information. This identifies the user's free time. Next, the server analyzes the user's social media data and other usage history to extract the user's interests and past activities. Based on this information, it selects activities that are likely to be highly satisfying for the user within the given time constraints.

[0221] The device's role is to notify the user of recommended activities received from the server. Notifications are customized according to the user's preferences, allowing for options such as push notifications or email. The user can review the notification and either add the suggested activity to their schedule or decline the suggestion.

[0222] When a user accepts or rejects an activity, the device sends this response as feedback to the server. The server uses this feedback to adjust its suggestion algorithm and improve the accuracy of future suggestions.

[0223] As a concrete example, consider a scenario where a user has an hour of free time during their lunch break. The server detects this free time and suggests an activity: visiting a nearby gallery that the user has previously expressed interest in on social media. The device notifies the user of this suggestion, and if the user accepts, the activity is automatically added to their schedule.

[0224] In this way, the present invention efficiently proposes activities that match the user's interests and needs within time constraints, thereby supporting a fulfilling lifestyle for the user.

[0225] The following describes the processing flow.

[0226] Step 1:

[0227] The server accesses the user's schedule database to retrieve the user's appointments. This allows it to identify available time slots and generate free time data.

[0228] Step 2:

[0229] The server collects data related to the user's past posts and interests through SNS APIs. Furthermore, it uses machine learning models to analyze activities and themes that the user is likely to be interested in.

[0230] Step 3:

[0231] The server combines free time data with user interest data to select the most suitable activity. The selection is optimized to suggest beneficial activities within the user's accessible range, taking into account their current location.

[0232] Step 4:

[0233] The device receives suggested activities from the server and displays a notification to the user. The notification format is determined according to the user's settings, and may include push notifications or email.

[0234] Step 5:

[0235] The user reviews the device notification and chooses to accept or reject the suggested activity. The user's choice is recorded by the device.

[0236] Step 6:

[0237] The terminal sends the user's selection results to the server. The server analyzes the results and adjusts the suggestion algorithm. By adjusting the algorithm, the accuracy of future suggestions is improved, and the user's preferences are reflected more accurately.

[0238] (Example 1)

[0239] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0240] In today's information-driven society, individuals have limited time to dedicate to enriching activities based on their hobbies and interests amidst their busy daily lives. Therefore, there is a need for a system that efficiently suggests activities suited to individual free time, thereby improving users' quality of life. However, conventional systems often fail to adequately reflect user interests in their suggestions, and improving the accuracy of real-time suggestions is challenging.

[0241] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0242] In this invention, the server includes means for acquiring the user's time information and identifying spatial availability, means for analyzing the user's interests and past behavioral history, and means for selecting activities that will provide high satisfaction to the user using a generating AI. This makes it possible to suggest activities that are customized to each individual user and are highly accurate and delivered in real time.

[0243] "User time information" refers to information about an individual's available time within a specific period, such as their schedule or appointments.

[0244] "Spatial leeway" refers to unscheduled free time within a user's schedule, providing a gap in time for engaging in new activities.

[0245] "Interest" refers to the matters that a user shows interest in or attention to regarding a particular theme or field.

[0246] "Past behavioral history" refers to a record of various activities and choices a user has made in the past, and serves as data for analyzing their interests and tendencies.

[0247] "Generative AI" refers to a model or method that uses artificial intelligence technology to automatically generate new data, information, or suggestions.

[0248] A "high-satisfaction activity" is an activity that users find interesting and that is considered to improve their quality of life.

[0249] "Information and communication technology" is a general term for technologies and methods for efficiently transmitting, receiving, and processing information.

[0250] "Digital data" refers to information in a format handled by computers and digital devices, which is generated, stored, and transmitted electronically.

[0251] An "activity suggestion algorithm" is a set of calculation procedures and rules used to select activities that are suitable for the user.

[0252] "Location information" is data that indicates a user's geographical location at a specific time, and is used to optimize activities.

[0253] In order to implement this invention, the server, terminal, and user each need to play a specific role.

[0254] The server uses an API from a cloud-based scheduling tool to retrieve the user's schedule information. This tool retrieves the user's schedule information as digital data and stores it in JSON format. Specifically, it is expected that scheduling tools such as the Google Calendar API will be used. The server processes this information to identify the user's time information.

[0255] Next, the server uses social media APIs to analyze the user's interests and past behavior history. For example, it retrieves user "likes" and follow information through the Twitter and Instagram APIs and analyzes it using the Python pandas library. This process identifies the user's areas of interest, and the results are used by the generative AI to select activities that will provide high satisfaction. The generative AI model used is one designed to generate activities based on the user's behavior.

[0256] An example of a prompt for the generating AI model is: "Consider the user's schedule and social media data, and suggest the most suitable activity for their next free time. Example: 1-hour lunch break, interest: art."

[0257] The device serves to notify the user of suggested activities. It uses push notification services such as Firebase Cloud Messaging to communicate selected activities to the user using information and communication technology. These notifications can be customized to the user's preferences, and email notifications are also an option.

[0258] Users can check notifications from their devices and choose to accept or reject suggested activities. This response is sent back to the server and recorded as digital data. Using this data, the server can adaptively improve the activity suggestion algorithm and increase the accuracy of future suggestions.

[0259] In this way, users, servers, and terminals work together to dynamically suggest activities that are optimal for each individual user, thereby improving the user's quality of life.

[0260] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0261] Step 1:

[0262] The server retrieves the user's schedule information. The input is the user's schedule data obtained from the API of a cloud-based scheduling tool. The server analyzes this data, uses a Python library to identify spatial availability, and identifies the next available time slots. The output is a list of available time slots. Specifically, the server requests schedule information using the API key and analyzes the returned data.

[0263] Step 2:

[0264] The server collects and analyzes user interests and past behavioral history. The input is user "likes" and follow information obtained through SNS APIs. The server uses the pandas library to organize the data and extract specific areas of interest. The output is a list of keywords indicating the user's interests. Specifically, the server sends API requests to SNS accounts to collect interest data.

[0265] Step 3:

[0266] The server uses a generative AI model to select activities that will provide high satisfaction to the user. The input consists of a list of free time slots and a list of keywords of the user's interests. The server inputs these as prompts into the generative AI model to generate the optimal activity. The output is the recommended activity. Specifically, the server inputs prompts into the generative AI and analyzes the results obtained.

[0267] Step 4:

[0268] The device notifies the user of recommended activities received from the server. The input is the recommended activities sent from the server. The device uses this information to send a push notification to the user. The output is the notification displayed on the user's device. Specifically, the device uses Firebase Cloud Messaging to send the notification.

[0269] Step 5:

[0270] The user checks the notification from their device and accepts or rejects the offered activity. The input is the notification from the device. The user's choice is recorded as an action and returned to the server as feedback. The output is the result of the user's choice. In concrete terms, the user inputs their choice through the interface on their device.

[0271] Step 6:

[0272] The server receives user selections as feedback and updates the activity suggestion algorithm. The input is the user's selection results. The server analyzes this data and adjusts the algorithm's weights. The resulting output is an updated algorithm designed to improve the accuracy of future suggestions. Specifically, the server saves the feedback to a database and retrains the algorithm.

[0273] (Application Example 1)

[0274] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0275] In modern times, many people find it difficult to manage their time in their daily lives, and there is a particular need to make efficient use of their free time. However, there are limited systems that can suggest meaningful activities tailored to individual lifestyles and interests, and these have not been able to increase user satisfaction. Furthermore, there is a need for suggestions using visual means that can provide more intuitive and immediate information.

[0276] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0277] In this invention, the server includes means for detecting the user's free time, means for extracting the user's usage history and interests, and means for presenting real-time activity information based on spatial circumstances via a visual information display device. This enables efficient and satisfying use of time by suggesting optimized activities to the user and notifying them of these suggestions visually and intuitively.

[0278] "Means for detecting a user's free time" refers to a function that analyzes a user's schedule information to identify time periods when normal activities are not scheduled.

[0279] The "means for extracting the user's usage history and interests" is a function that analyzes past behavior data and records of electronic information exchange to grasp the matters and behavioral tendencies that the user is interested in.

[0280] The "means for proposing activities that are highly satisfactory to the user" is a function that selects optimal activities based on the user's free time and interests and makes recommendations to enhance satisfaction.

[0281] The "means for notifying the user of the proposal" is a means of conveying the selected activity to the user, and is a function of displaying or informing the user's terminal of information using visual or auditory methods.

[0282] The "means for collecting the user's reactions and updating the proposal model" is a function that accumulates the user's responses to the proposal and continuously improves the algorithm for activity proposals based on that data.

[0283] The "means for presenting real-time activity information based on the spatial situation via a visual information display device" is a function that visually shows the user activities according to real-time position information and the surrounding environment, and concretizes the information.

[0284] This system is for effectively utilizing the user's free time and supporting a highly satisfactory life. The server acquires the user's schedule information and identifies free time. In this process, it analyzes the schedule information stored in the database management system on the cloud. Next, the server analyzes SNS data and data related to past electronic information exchange using a machine learning model to extract the user's usage history and interests.

[0285] The terminal is a visual information display device such as smart glasses, and notifies the user of the optimal activity proposal received from the server. This notification includes real-time proposals considering the spatial situation based on the user's current position information. For example, when the user is walking in the city, information about an event being held in a nearby park is displayed. With this intuitive display, the user can consciously select that activity.

[0286] Furthermore, the user's reaction is always fed back from the terminal to the server, contributing to the improvement of the accuracy of the proposed algorithm. Learning is performed using the AI model generated by this feedback, and the proposals for subsequent times are adjusted to better match the user's needs.

[0287] To give a specific example, when the user is at a café during the lunch break on weekdays, the server recommends beneficial activities taking place in the vicinity based on the location. An example of the prompt sentence input into the generated AI model is "Please propose activities suitable for the current location based on the user's schedule and SNS data." Through this series of processes, it is possible to enrich the user's life.

[0288] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0289] Step 1:

[0290] The server obtains the user's schedule data from the cloud database. The input is the user's schedule information, and based on this, free time is identified. As a specific operation, the database management system is used to access and analyze the user's calendar data to detect time periods without a schedule.

[0291] Step 2:

[0292] The server obtains SNS data and past electronic information history, and extracts the user's interests using the generated AI model. The inputs are SNS data and usage history, and the output is the user's areas of concern. The specific operation is to analyze the text data using natural language processing technology and list the user's areas of interest.

[0293] Step 3:

[0294] Based on the above information, the server selects suitable activities and generates activity information optimized in real time based on the user's current location. Inputs include free time, interests, and location information, while output is activity suggestions. The server uses location services to search, select, and recommend events and activities relevant to the user's location.

[0295] Step 4:

[0296] The smart glasses, acting as the terminal, receive suggestions from the server and present them to the user as visual notifications. The input is activity suggestions from the server, and the output is a visual display. Specific actions include displaying information on the screen in a conspicuous manner through the user interface.

[0297] Step 5:

[0298] The user reviews the suggested activity and responds by selecting or rejecting it. The input is a visual notification, and the output is the user's choice. The user looks at the displayed suggestions, makes a selection based on their interests, and sends feedback to the server via the terminal.

[0299] Step 6:

[0300] The server receives user feedback and updates the proposed algorithm. The input is the user's selection and feedback, and the output is the updated proposed model. Specifically, the feedback is input into the machine learning algorithm to tune the model and improve the accuracy of future proposals.

[0301] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0302] The system of the present invention is an activity recommendation system incorporating an emotion engine to efficiently utilize the user's free time and improve user satisfaction. This system recognizes the user's emotional state in real time and adjusts the recommended activities based on that information.

[0303] The server obtains the user's schedule information and identifies free time. Next, the server analyzes the user's SNS data and past usage history to extract interests. In this process, the emotion engine analyzes the user's emotional state. Emotions are evaluated in real time by text analysis and image recognition technologies. Based on this evaluation result, the server selects the activities to be recommended.

[0304] As a specific example, consider the case of recommending activities that can relax the user when they are feeling stressed. The server detects signs of stress from the user's facial expressions and text information, and recommends the optimal relaxation spots based on the user's time and interests. For example, it recommends taking a walk in a nearby park or reading a book in a quiet café.

[0305] The terminal receives the recommendation from the server and notifies the user. The notification is adjusted according to the user's settings. For example, priority is given to activities for stress relief. The user checks the notification on the terminal and, by accepting the recommendation, it is automatically added to the schedule.

[0306] Furthermore, by recording the user's reaction, the server can update the recommendation algorithm and be able to recommend more effective activities in the future. It is a system that can provide a dynamic and personalized experience according to the user's psychological state by utilizing the data of the emotion engine.

[0307] The processing flow will be described below.

[0308] Step 1:

[0309] The server retrieves user schedule data from the database. This identifies the user's current appointments and free time, and prepares the data for analysis.

[0310] Step 2:

[0311] The server collects users' past posts and activity history through APIs from social media and other platforms. The collected data is then analyzed using text to extract user interests and preferences.

[0312] Step 3:

[0313] The emotion engine identifies emotions from collected content to determine the user's situation. It uses keywords in text, as well as data obtained from audio and images, to assess the current emotional state.

[0314] Step 4:

[0315] Based on the interests and emotions identified by the server, the most suitable activities are selected. For example, for a user experiencing stress, activities that contribute to relaxation are prioritized.

[0316] Step 5:

[0317] The device receives recommended activities from the server and displays them to the user as notifications. These notifications are presented prominently within the user interface and designed to attract the user's attention.

[0318] Step 6:

[0319] Users review notifications and accept or reject suggested activities. If accepted, the activity is automatically added to the user's schedule, reducing the effort required for discovery.

[0320] Step 7:

[0321] The device sends the user's selection as feedback to the server. This feedback is used to refine the algorithm and improve the accuracy of future suggestions.

[0322] This process enables personalized activity suggestions that are tailored to the user's emotions and preferences.

[0323] (Example 2)

[0324] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0325] In modern society, many people are unaware of their own emotional state due to the busyness and stress of daily life, and as a result, are unable to effectively utilize their free time. There is a need to improve user satisfaction and reduce stress by suggesting optimal activities based on the user's emotional state and interests.

[0326] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0327] In this invention, the server includes means for detecting the user's free time, means for analyzing the user's emotional state in real time, and means for extracting the user's interests based on the information. This makes it possible for the user to make the most of their free time in the most optimal way.

[0328] "Free time" refers to periods in a user's schedule when no specific appointments are scheduled.

[0329] "Emotional state" refers to information that indicates the user's current psychological or emotional condition, and is derived from text and images.

[0330] "Real-time analysis" refers to processing user data instantly without any time delay and obtaining results immediately.

[0331] "Interest extraction" refers to identifying themes and activities that a user is interested in, based on their past behavior and digital information.

[0332] "Activity suggestions" refer to recommending specific actions or activities to users based on their emotional state and interests.

[0333] "Notifications" refer to sending messages or alerts to inform users about suggested activities.

[0334] "Response recording" refers to saving data on the actions and choices a user makes in response to suggested activities.

[0335] "Updating the suggestion algorithm" means improving the algorithm based on recorded user feedback to enhance the accuracy of future suggestions.

[0336] The present invention is a system that uses multiple computer resources to provide activity suggestions that take into account the user's emotional state. The server first obtains the user's schedule information using a specific API and identifies available time. In this process, data is collected, for example, from a calendar application.

[0337] Next, the server uses natural language processing tools (e.g., spaCy and NLTK) to analyze the user's social media data and past usage history, extracting the user's interests. It also utilizes the Sentiment Analysis API and image recognition technologies (e.g., OpenCV and Amazon Rekognition) as sentiment engines to evaluate the user's emotional state in real time from their text information and photos. This evaluation provides the foundational data needed to guide appropriate activities.

[0338] The server utilizes the user's current location information and leverages geographic information services (e.g., Google Maps API) to optimize activity suggestions. This allows it to select activities available near the user and suggest relaxing spots and entertainment options tailored to the user's emotional state.

[0339] The device is responsible for notifying the user of suggestions from the server. Notifications are delivered via push notifications or email according to the user's preferences, and features such as prioritizing activities that take emotional state into consideration are implemented. When the user accepts a suggestion, it is automatically added to the schedule.

[0340] For example, if the server determines from a user's social media posts or images that they are "feeling stressed," it can suggest a walk in a nearby park or a quiet cafe. The AI ​​model generates prompts like the following to suggest activities that can help the user relax: "Please suggest activities that can help the user relax when they are feeling stressed."

[0341] User responses are recorded, and this data is analyzed by the server and used to update the proposed algorithm. This feedback allows the system to provide optimal, personalized activities for the user over the long term.

[0342] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0343] Step 1:

[0344] The server retrieves the user's schedule information. The input is data from the user's calendar application. The server analyzes this information to identify free time slots that are not scheduled. The output is a list of the user's next free time slots. Specifically, the server retrieves data from the calendar application via an API and determines whether there are appointments or not for each date and time.

[0345] Step 2:

[0346] The server analyzes the user's social media data and past usage history to extract their interests. The input consists of the user's past web browsing history and social media posts. The server processes this data using natural language processing tools to extract specific keywords and topics. The output is a list of keywords indicating the user's interests. Specifically, the server uses a text analysis engine to evaluate frequently occurring words and contexts to identify areas of interest.

[0347] Step 3:

[0348] The server evaluates the user's real-time emotional state. The input is the text and images from the user's latest social media posts. The server uses an emotion engine to analyze the sentiment of the text and images and determine the user's current emotional state. The output is a positive, negative, or neutral emotional state. Specifically, the server uses an emotion analysis API to aggregate positive and negative scores for words, and also uses image recognition to infer emotions from facial expressions.

[0349] Step 4:

[0350] The server selects appropriate activities based on the user's free time, interests, and emotional state. The inputs are free time, user interests, and emotional state. The server uses a generative AI model to generate the optimal activity from this data. The output is the activity suggested to the user. Specifically, the server dynamically generates different activity candidates, evaluates them, and selects the most suitable one.

[0351] Step 5:

[0352] The device notifies the user of selected activity suggestions. The input is the activity suggestion sent from the server. The device sends a push notification to deliver this information to the user. The output is the activity suggestion message displayed on the user's device. Specifically, the device activates the notification system and delivers a message to the user accompanied by an alarm or vibration depending on the importance level.

[0353] Step 6:

[0354] The user reacts to the suggested activity, and the result is recorded. The input is the user's reaction, i.e., their acceptance or rejection of the suggestion. The server stores this information in a reaction log and uses it to improve the suggestion algorithm. The output is the updated algorithm. Specifically, the server periodically takes the user's selection history as training data and forms a feedback loop to improve the accuracy of suggestions in the future.

[0355] (Application Example 2)

[0356] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the smart glasses 214 as the "terminal".

[0357] In modern society, many users experience stress and fatigue in their daily lives. Furthermore, it is difficult to suggest optimal activities based on users' emotional states and interests amidst their busy schedules. Therefore, this invention aims to improve user satisfaction by efficiently utilizing users' free time and suggesting activities that match their emotional state.

[0358] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0359] In this invention, the server includes means for detecting the user's free time, means for extracting the user's usage history and interests, and means for analyzing the user's emotional state using an emotion recognition engine and adjusting the suggested activities. This makes it possible to suggest appropriate activities to the user in real time according to their emotional state.

[0360] "User free time" refers to the period of time when a user has no scheduled appointments and is free to use as they please.

[0361] "Usage history" refers to data that records information about activities and services a user has used in the past.

[0362] "Interest" refers to information related to the subjects or actions that users are interested in.

[0363] An "emotion recognition engine" refers to a program or system that analyzes a user's emotional state from their facial expressions, voice, and text data.

[0364] "Real-time evaluation" refers to a process that analyzes and outputs results instantly at the moment data is generated.

[0365] "Activities" refer to actions, tasks, or activities that become part of a user's life, such as relaxation.

[0366] "Notifications" refer to sending messages or alerts to inform users of information or suggestions.

[0367] "Response" refers to the user's reply or action in response to a suggestion or notification.

[0368] A "proposal model" refers to an algorithm or mechanism for selecting content to propose to the user.

[0369] In this embodiment of the invention, the system primarily consists of a server and terminals. The server has the capability to detect users' free time and process large amounts of data to extract usage history and interests. This includes cloud-based data storage and an analysis engine. The server also analyzes the user's real-time emotional state using an emotion recognition engine. To this end, it uses OpenCV for image recognition technology and a virtual emotion recognition library for emotion analysis.

[0370] The device functions as a user interface, notifying the user of activity suggestions. These notifications can be customized according to user settings; for example, suggestions aimed at stress relief can be prioritized. The device functions as a smartphone or consumer robot, recording user responses and sending feedback to a server.

[0371] This feedback loop allows the suggestion model to be continuously updated and evolve to recommend even more personalized activities to the user. For example, if the device determines that a user is stressed upon returning home, it may play soothing music to create a relaxing environment.

[0372] An example of a prompt suitable for a generative AI model is: "Analyze the user's facial expressions from the image captured by the camera, and if stress is detected, suggest a relaxing activity."

[0373] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0374] Step 1:

[0375] The server retrieves user schedule information and identifies available time slots. The input is the user's schedule data, and the output is the identified available time slots. This process fetches schedules from a database and extracts available time slots using an algorithm.

[0376] Step 2:

[0377] The server analyzes the user's social media data and past usage history to extract their interests. Input is social media posts and past activity logs, and output is information about the user's interests. Data analysis tools are used, and text mining techniques are employed to identify areas of interest.

[0378] Step 3:

[0379] The server uses an emotion recognition engine to analyze the user's emotional state in real time. Input is the user's facial image and voice data, and output is the analyzed emotional state. It utilizes OpenCV and a virtual emotion recognition library to scan facial expressions and voice patterns.

[0380] Step 4:

[0381] The server selects appropriate activities based on the user's free time, interests, and emotional state. The input is free time, interests, and emotional information, and the output is suggested activities. An algorithm is used to generate recommended activities by considering multiple factors.

[0382] Step 5:

[0383] The device receives suggestions from the server and notifies the user. The input is suggestion activity information, and the output is the notification to the user. The notification is displayed on the device as a pop-up or voice guidance.

[0384] Step 6:

[0385] The user checks the notification on their device and selects or rejects an activity based on it. The input is the suggested activity, and the output is the user's response. By selecting an option, the user determines their next action.

[0386] Step 7:

[0387] The server records user responses and updates the suggestion model. The input is user response data, and the output is the updated model. This allows future suggestions to be more personalized and the model to better suit the user's preferences.

[0388] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0389] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0390] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0391] [Third Embodiment]

[0392] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0393] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0394] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0395] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0396] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0397] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0398] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0399] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0400] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0401] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0402] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0403] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0404] The system of this invention is designed to enable users to efficiently utilize their time in their daily lives and engage in highly satisfying activities. This system improves the user's quality of life by detecting their free time and suggesting activities that match that time.

[0405] The server plays a central role, first obtaining the user's schedule information. This identifies the user's free time. Next, the server analyzes the user's social media data and other usage history to extract the user's interests and past activities. Based on this information, it selects activities that are likely to be highly satisfying for the user within the given time constraints.

[0406] The device's role is to notify the user of recommended activities received from the server. Notifications are customized according to the user's preferences, allowing for options such as push notifications or email. The user can review the notification and either add the suggested activity to their schedule or decline the suggestion.

[0407] When a user accepts or rejects an activity, the device sends this response as feedback to the server. The server uses this feedback to adjust its suggestion algorithm and improve the accuracy of future suggestions.

[0408] As a concrete example, consider a scenario where a user has an hour of free time during their lunch break. The server detects this free time and suggests an activity: visiting a nearby gallery that the user has previously expressed interest in on social media. The device notifies the user of this suggestion, and if the user accepts, the activity is automatically added to their schedule.

[0409] In this way, the present invention efficiently proposes activities that match the user's interests and needs within time constraints, thereby supporting a fulfilling lifestyle for the user.

[0410] The following describes the processing flow.

[0411] Step 1:

[0412] The server accesses the user's schedule database to retrieve the user's appointments. This allows it to identify available time slots and generate free time data.

[0413] Step 2:

[0414] The server collects data related to the user's past posts and interests through SNS APIs. Furthermore, it uses machine learning models to analyze activities and themes that the user is likely to be interested in.

[0415] Step 3:

[0416] The server combines free time data with user interest data to select the most suitable activity. The selection is optimized to suggest beneficial activities within the user's accessible range, taking into account their current location.

[0417] Step 4:

[0418] The device receives suggested activities from the server and displays a notification to the user. The notification format is determined according to the user's settings, and may include push notifications or email.

[0419] Step 5:

[0420] The user reviews the device notification and chooses to accept or reject the suggested activity. The user's choice is recorded by the device.

[0421] Step 6:

[0422] The terminal sends the user's selection results to the server. The server analyzes the results and adjusts the suggestion algorithm. By adjusting the algorithm, the accuracy of future suggestions is improved, and the user's preferences are reflected more accurately.

[0423] (Example 1)

[0424] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0425] In today's information-driven society, individuals have limited time to dedicate to enriching activities based on their hobbies and interests amidst their busy daily lives. Therefore, there is a need for a system that efficiently suggests activities suited to individual free time, thereby improving users' quality of life. However, conventional systems often fail to adequately reflect user interests in their suggestions, and improving the accuracy of real-time suggestions is challenging.

[0426] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0427] In this invention, the server includes means for acquiring the user's time information and identifying spatial availability, means for analyzing the user's interests and past behavioral history, and means for selecting activities that will provide high satisfaction to the user using a generating AI. This makes it possible to suggest activities that are customized to each individual user and are highly accurate and delivered in real time.

[0428] "User time information" refers to information about an individual's available time within a specific period, such as their schedule or appointments.

[0429] "Spatial leeway" refers to unscheduled free time within a user's schedule, providing a gap in time for engaging in new activities.

[0430] "Interest" refers to the matters that a user shows interest in or attention to regarding a particular theme or field.

[0431] "Past behavioral history" refers to a record of various activities and choices a user has made in the past, and serves as data for analyzing their interests and tendencies.

[0432] "Generative AI" refers to a model or method that uses artificial intelligence technology to automatically generate new data, information, or suggestions.

[0433] A "high-satisfaction activity" is an activity that users find interesting and that is considered to improve their quality of life.

[0434] "Information and communication technology" is a general term for technologies and methods for efficiently transmitting, receiving, and processing information.

[0435] "Digital data" refers to information in a format handled by computers and digital devices, which is generated, stored, and transmitted electronically.

[0436] An "activity suggestion algorithm" is a set of calculation procedures and rules used to select activities that are suitable for the user.

[0437] "Location information" is data that indicates a user's geographical location at a specific time, and is used to optimize activities.

[0438] In order to implement this invention, the server, terminal, and user each need to play a specific role.

[0439] The server uses an API from a cloud-based scheduling tool to retrieve the user's schedule information. This tool retrieves the user's schedule information as digital data and stores it in JSON format. Specifically, it is expected that scheduling tools such as the Google Calendar API will be used. The server processes this information to identify the user's time information.

[0440] Next, the server uses social media APIs to analyze the user's interests and past behavior history. For example, it retrieves user "likes" and follow information through the Twitter and Instagram APIs and analyzes it using the Python pandas library. This process identifies the user's areas of interest, and the results are used by the generative AI to select activities that will provide high satisfaction. The generative AI model used is one designed to generate activities based on the user's behavior.

[0441] An example of a prompt for the generating AI model is: "Consider the user's schedule and social media data, and suggest the most suitable activity for their next free time. Example: 1-hour lunch break, interest: art."

[0442] The device serves to notify the user of suggested activities. It uses push notification services such as Firebase Cloud Messaging to communicate selected activities to the user using information and communication technology. These notifications can be customized to the user's preferences, and email notifications are also an option.

[0443] Users can check notifications from their devices and choose to accept or reject suggested activities. This response is sent back to the server and recorded as digital data. Using this data, the server can adaptively improve the activity suggestion algorithm and increase the accuracy of future suggestions.

[0444] In this way, users, servers, and terminals work together to dynamically suggest activities that are optimal for each individual user, thereby improving the user's quality of life.

[0445] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0446] Step 1:

[0447] The server retrieves the user's schedule information. The input is the user's schedule data obtained from the API of a cloud-based scheduling tool. The server analyzes this data, uses a Python library to identify spatial availability, and identifies the next available time slots. The output is a list of available time slots. Specifically, the server requests schedule information using the API key and analyzes the returned data.

[0448] Step 2:

[0449] The server collects and analyzes user interests and past behavioral history. The input is user "likes" and follow information obtained through SNS APIs. The server uses the pandas library to organize the data and extract specific areas of interest. The output is a list of keywords indicating the user's interests. Specifically, the server sends API requests to SNS accounts to collect interest data.

[0450] Step 3:

[0451] The server uses a generative AI model to select activities that will provide high satisfaction to the user. The input consists of a list of free time slots and a list of keywords of the user's interests. The server inputs these as prompts into the generative AI model to generate the optimal activity. The output is the recommended activity. Specifically, the server inputs prompts into the generative AI and analyzes the results obtained.

[0452] Step 4:

[0453] The device notifies the user of recommended activities received from the server. The input is the recommended activities sent from the server. The device uses this information to send a push notification to the user. The output is the notification displayed on the user's device. Specifically, the device uses Firebase Cloud Messaging to send the notification.

[0454] Step 5:

[0455] The user checks the notification from their device and accepts or rejects the offered activity. The input is the notification from the device. The user's choice is recorded as an action and returned to the server as feedback. The output is the result of the user's choice. In concrete terms, the user inputs their choice through the interface on their device.

[0456] Step 6:

[0457] The server receives user selections as feedback and updates the activity suggestion algorithm. The input is the user's selection results. The server analyzes this data and adjusts the algorithm's weights. The resulting output is an updated algorithm designed to improve the accuracy of future suggestions. Specifically, the server saves the feedback to a database and retrains the algorithm.

[0458] (Application Example 1)

[0459] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0460] In modern times, many people find it difficult to manage their time in their daily lives, and there is a particular need to make efficient use of their free time. However, there are limited systems that can suggest meaningful activities tailored to individual lifestyles and interests, and these have not been able to increase user satisfaction. Furthermore, there is a need for suggestions using visual means that can provide more intuitive and immediate information.

[0461] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0462] In this invention, the server includes means for detecting the user's free time, means for extracting the user's usage history and interests, and means for presenting real-time activity information based on spatial circumstances via a visual information display device. This enables efficient and satisfying use of time by suggesting optimized activities to the user and notifying them of these suggestions visually and intuitively.

[0463] "Means for detecting a user's free time" refers to a function that analyzes a user's schedule information to identify time periods when normal activities are not scheduled.

[0464] "Means for extracting user usage history and interests" refers to functions that analyze past behavioral data and records of electronic information exchange to understand the topics of interest and behavioral trends of users.

[0465] "A means of suggesting highly satisfying activities to users" refers to a function that selects the most suitable activities based on the user's free time and interests, and makes recommendations to increase satisfaction.

[0466] "Means of notifying users of suggestions" refers to a means of informing users of selected activities, which is a function that displays or notifies users of information on their devices using visual or auditory methods.

[0467] "Means for collecting user responses and updating the suggestion model" refers to a function that accumulates user responses to suggestions and continuously improves the activity suggestion algorithm based on that data.

[0468] "Means for presenting real-time activity information based on spatial circumstances via a visual information display device" refers to a function that visually displays real-time location information and activities corresponding to the surrounding environment to the user, thereby concretizing the information.

[0469] This system is designed to help users make effective use of their free time and support a more fulfilling lifestyle. The server retrieves the user's schedule information and identifies their free time. This process involves analyzing schedule information stored in a cloud-based database management system. Next, the server uses machine learning models to analyze SNS data and past electronic information exchange data to extract the user's usage history and interests.

[0470] The device is a visual information display device, such as smart glasses, that notifies the user of optimal activity suggestions received from a server. These suggestions include real-time recommendations that take spatial context into account, based on the user's current location. For example, while a user is walking in the city, information about events taking place in a nearby park might be displayed. This intuitive display allows the user to consciously choose their activity.

[0471] Furthermore, user feedback is constantly fed back from the device to the server, contributing to the improvement of the proposed algorithm's accuracy. This feedback is used to train the generative AI model, which then adjusts subsequent proposals to better match user needs.

[0472] For example, if a user is at a cafe during their lunch break on a weekday, the server will recommend beneficial activities in the vicinity based on their location. An example of a prompt to input into the generating AI model would be, "Based on the user's schedule and social media data, please suggest activities that are relevant to their current location." Through this entire process, it is possible to enrich the user's life.

[0473] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0474] Step 1:

[0475] The server retrieves user schedule data from a cloud database. The input is the user's schedule information, which is used to identify free time. Specifically, it uses a database management system to access and analyze the user's calendar data to detect periods of free time.

[0476] Step 2:

[0477] The server acquires SNS data and past electronic information history, and uses a generative AI model to extract user interests. The input is SNS data and usage history, and the output is the user's areas of interest. Specifically, it analyzes text data using natural language processing techniques and lists the user's areas of interest.

[0478] Step 3:

[0479] Based on the above information, the server selects suitable activities and generates activity information optimized in real time based on the user's current location. Inputs include free time, interests, and location information, while output is activity suggestions. The server uses location services to search, select, and recommend events and activities relevant to the user's location.

[0480] Step 4:

[0481] The smart glasses, acting as the terminal, receive suggestions from the server and present them to the user as visual notifications. The input is activity suggestions from the server, and the output is a visual display. Specific actions include displaying information on the screen in a conspicuous manner through the user interface.

[0482] Step 5:

[0483] The user reviews the suggested activity and responds by selecting or rejecting it. The input is a visual notification, and the output is the user's choice. The user looks at the displayed suggestions, makes a selection based on their interests, and sends feedback to the server via the terminal.

[0484] Step 6:

[0485] The server receives user feedback and updates the proposed algorithm. The input is the user's selection and feedback, and the output is the updated proposed model. Specifically, the feedback is input into the machine learning algorithm to tune the model and improve the accuracy of future proposals.

[0486] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0487] The present invention is an activity suggestion system that incorporates an emotion engine to efficiently utilize users' free time and improve user satisfaction. This system recognizes the user's emotional state in real time and adjusts the suggested activities based on that information.

[0488] The server retrieves the user's schedule information and identifies their free time. Next, the server analyzes the user's social media data and past usage history to extract their interests. During this process, the emotion engine analyzes the user's emotional state. Emotions are evaluated in real time using text analysis and image recognition technologies. Based on these evaluation results, the server selects activities to suggest.

[0489] As a concrete example, consider a scenario where a user is feeling stressed and needs to be offered relaxing activities. The server detects signs of stress from the user's facial expressions and text information, and then suggests the most suitable relaxation spot based on the user's time and interests. For example, it might recommend a walk in a nearby park or reading in a quiet cafe.

[0490] The device receives suggestions from the server and notifies the user. Notifications are tailored to the user's settings, with priority given to activities such as stress relief. The user checks the device's notifications and accepts the suggestions, which are automatically added to the schedule.

[0491] Furthermore, by recording user responses, the server can update its suggestion algorithm, enabling it to propose more effective activities in the future. This system leverages emotion engine data to provide a dynamic and personalized experience tailored to the user's psychological state.

[0492] The following describes the processing flow.

[0493] Step 1:

[0494] The server retrieves user schedule data from the database. This identifies the user's current appointments and free time, and prepares the data for analysis.

[0495] Step 2:

[0496] The server collects users' past posts and activity history through APIs from social media and other platforms. The collected data is then analyzed using text to extract user interests and preferences.

[0497] Step 3:

[0498] The emotion engine identifies emotions from collected content to determine the user's situation. It uses keywords in text, as well as data obtained from audio and images, to assess the current emotional state.

[0499] Step 4:

[0500] Based on the interests and emotions identified by the server, the most suitable activities are selected. For example, for a user experiencing stress, activities that contribute to relaxation are prioritized.

[0501] Step 5:

[0502] The device receives recommended activities from the server and displays them to the user as notifications. These notifications are presented prominently within the user interface and designed to attract the user's attention.

[0503] Step 6:

[0504] Users review notifications and accept or reject suggested activities. If accepted, the activity is automatically added to the user's schedule, reducing the effort required for discovery.

[0505] Step 7:

[0506] The device sends the user's selection as feedback to the server. This feedback is used to refine the algorithm and improve the accuracy of future suggestions.

[0507] This process enables personalized activity suggestions that are tailored to the user's emotions and preferences.

[0508] (Example 2)

[0509] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0510] In modern society, many people are unaware of their own emotional state due to the busyness and stress of daily life, and as a result, are unable to effectively utilize their free time. There is a need to improve user satisfaction and reduce stress by suggesting optimal activities based on the user's emotional state and interests.

[0511] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0512] In this invention, the server includes means for detecting the user's free time, means for analyzing the user's emotional state in real time, and means for extracting the user's interests based on the information. This makes it possible for the user to make the most of their free time in the most optimal way.

[0513] "Free time" refers to periods in a user's schedule when no specific appointments are scheduled.

[0514] "Emotional state" refers to information that indicates the user's current psychological or emotional condition, and is derived from text and images.

[0515] "Real-time analysis" refers to processing user data instantly without any time delay and obtaining results immediately.

[0516] "Interest extraction" refers to identifying themes and activities that a user is interested in, based on their past behavior and digital information.

[0517] "Activity suggestions" refer to recommending specific actions or activities to users based on their emotional state and interests.

[0518] "Notifications" refer to sending messages or alerts to inform users about suggested activities.

[0519] "Response recording" refers to saving data on the actions and choices a user makes in response to suggested activities.

[0520] "Updating the suggestion algorithm" means improving the algorithm based on recorded user feedback to enhance the accuracy of future suggestions.

[0521] The present invention is a system that uses multiple computer resources to provide activity suggestions that take into account the user's emotional state. The server first obtains the user's schedule information using a specific API and identifies available time. In this process, data is collected, for example, from a calendar application.

[0522] Next, the server uses natural language processing tools (e.g., spaCy and NLTK) to analyze the user's social media data and past usage history, extracting the user's interests. It also utilizes the Sentiment Analysis API and image recognition technologies (e.g., OpenCV and Amazon Rekognition) as sentiment engines to evaluate the user's emotional state in real time from their text information and photos. This evaluation provides the foundational data needed to guide appropriate activities.

[0523] The server utilizes the user's current location information and leverages geographic information services (e.g., Google Maps API) to optimize activity suggestions. This allows it to select activities available near the user and suggest relaxing spots and entertainment options tailored to the user's emotional state.

[0524] The device is responsible for notifying the user of suggestions from the server. Notifications are delivered via push notifications or email according to the user's preferences, and features such as prioritizing activities that take emotional state into consideration are implemented. When the user accepts a suggestion, it is automatically added to the schedule.

[0525] For example, if the server determines from a user's social media posts or images that they are "feeling stressed," it can suggest a walk in a nearby park or a quiet cafe. The AI ​​model generates prompts like the following to suggest activities that can help the user relax: "Please suggest activities that can help the user relax when they are feeling stressed."

[0526] User responses are recorded, and this data is analyzed by the server and used to update the proposed algorithm. This feedback allows the system to provide optimal, personalized activities for the user over the long term.

[0527] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0528] Step 1:

[0529] The server retrieves the user's schedule information. The input is data from the user's calendar application. The server analyzes this information to identify free time slots that are not scheduled. The output is a list of the user's next free time slots. Specifically, the server retrieves data from the calendar application via an API and determines whether there are appointments or not for each date and time.

[0530] Step 2:

[0531] The server analyzes the user's social media data and past usage history to extract their interests. The input consists of the user's past web browsing history and social media posts. The server processes this data using natural language processing tools to extract specific keywords and topics. The output is a list of keywords indicating the user's interests. Specifically, the server uses a text analysis engine to evaluate frequently occurring words and contexts to identify areas of interest.

[0532] Step 3:

[0533] The server evaluates the user's real-time emotional state. The input is the text and images from the user's latest social media posts. The server uses an emotion engine to analyze the sentiment of the text and images and determine the user's current emotional state. The output is a positive, negative, or neutral emotional state. Specifically, the server uses an emotion analysis API to aggregate positive and negative scores for words, and also uses image recognition to infer emotions from facial expressions.

[0534] Step 4:

[0535] The server selects appropriate activities based on the user's free time, interests, and emotional state. The inputs are free time, user interests, and emotional state. The server uses a generative AI model to generate the optimal activity from this data. The output is the activity suggested to the user. Specifically, the server dynamically generates different activity candidates, evaluates them, and selects the most suitable one.

[0536] Step 5:

[0537] The device notifies the user of selected activity suggestions. The input is the activity suggestion sent from the server. The device sends a push notification to deliver this information to the user. The output is the activity suggestion message displayed on the user's device. Specifically, the device activates the notification system and delivers a message to the user accompanied by an alarm or vibration depending on the importance level.

[0538] Step 6:

[0539] The user reacts to the suggested activity, and the result is recorded. The input is the user's reaction, i.e., their acceptance or rejection of the suggestion. The server stores this information in a reaction log and uses it to improve the suggestion algorithm. The output is the updated algorithm. Specifically, the server periodically takes the user's selection history as training data and forms a feedback loop to improve the accuracy of suggestions in the future.

[0540] (Application Example 2)

[0541] Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0542] In modern society, many users experience stress and fatigue in their daily lives. Furthermore, it is difficult to suggest optimal activities based on users' emotional states and interests amidst their busy schedules. Therefore, this invention aims to improve user satisfaction by efficiently utilizing users' free time and suggesting activities that match their emotional state.

[0543] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0544] In this invention, the server includes means for detecting the user's free time, means for extracting the user's usage history and interests, and means for analyzing the user's emotional state using an emotion recognition engine and adjusting the suggested activities. This makes it possible to suggest appropriate activities to the user in real time according to their emotional state.

[0545] "User free time" refers to the period of time when a user has no scheduled appointments and is free to use as they please.

[0546] "Usage history" refers to data that records information about activities and services a user has used in the past.

[0547] "Interest" refers to information related to the subjects or actions that users are interested in.

[0548] An "emotion recognition engine" refers to a program or system that analyzes a user's emotional state from their facial expressions, voice, and text data.

[0549] "Real-time evaluation" refers to a process that analyzes and outputs results instantly at the moment data is generated.

[0550] "Activities" refer to actions, tasks, or activities that become part of a user's life, such as relaxation.

[0551] "Notifications" refer to sending messages or alerts to inform users of information or suggestions.

[0552] "Response" refers to the user's reply or action in response to a suggestion or notification.

[0553] A "proposal model" refers to an algorithm or mechanism for selecting content to propose to the user.

[0554] In this embodiment of the invention, the system primarily consists of a server and terminals. The server has the capability to detect users' free time and process large amounts of data to extract usage history and interests. This includes cloud-based data storage and an analysis engine. The server also analyzes the user's real-time emotional state using an emotion recognition engine. To this end, it uses OpenCV for image recognition technology and a virtual emotion recognition library for emotion analysis.

[0555] The device functions as a user interface, notifying the user of activity suggestions. These notifications can be customized according to user settings; for example, suggestions aimed at stress relief can be prioritized. The device functions as a smartphone or consumer robot, recording user responses and sending feedback to a server.

[0556] This feedback loop allows the suggestion model to be continuously updated and evolve to recommend even more personalized activities to the user. For example, if the device determines that a user is stressed upon returning home, it may play soothing music to create a relaxing environment.

[0557] An example of a prompt suitable for a generative AI model is: "Analyze the user's facial expressions from the image captured by the camera, and if stress is detected, suggest a relaxing activity."

[0558] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0559] Step 1:

[0560] The server retrieves user schedule information and identifies available time slots. The input is the user's schedule data, and the output is the identified available time slots. This process fetches schedules from a database and extracts available time slots using an algorithm.

[0561] Step 2:

[0562] The server analyzes the user's social media data and past usage history to extract their interests. Input is social media posts and past activity logs, and output is information about the user's interests. Data analysis tools are used, and text mining techniques are employed to identify areas of interest.

[0563] Step 3:

[0564] The server uses an emotion recognition engine to analyze the user's emotional state in real time. Input is the user's facial image and voice data, and output is the analyzed emotional state. It utilizes OpenCV and a virtual emotion recognition library to scan facial expressions and voice patterns.

[0565] Step 4:

[0566] The server selects appropriate activities based on the user's free time, interests, and emotional state. The input is free time, interests, and emotional information, and the output is suggested activities. An algorithm is used to generate recommended activities by considering multiple factors.

[0567] Step 5:

[0568] The device receives suggestions from the server and notifies the user. The input is suggestion activity information, and the output is the notification to the user. The notification is displayed on the device as a pop-up or voice guidance.

[0569] Step 6:

[0570] The user checks the notification on their device and selects or rejects an activity based on it. The input is the suggested activity, and the output is the user's response. By selecting an option, the user determines their next action.

[0571] Step 7:

[0572] The server records user responses and updates the suggestion model. The input is user response data, and the output is the updated model. This allows future suggestions to be more personalized and the model to better suit the user's preferences.

[0573] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0574] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0575] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0576] [Fourth Embodiment]

[0577] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0578] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0579] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0580] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0581] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0582] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0583] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0584] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0585] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0586] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0587] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0588] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0589] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0590] The system of this invention is designed to enable users to efficiently utilize their time in their daily lives and engage in highly satisfying activities. This system improves the user's quality of life by detecting their free time and suggesting activities that match that time.

[0591] The server plays a central role, first obtaining the user's schedule information. This identifies the user's free time. Next, the server analyzes the user's social media data and other usage history to extract the user's interests and past activities. Based on this information, it selects activities that are likely to be highly satisfying for the user within the given time constraints.

[0592] The device's role is to notify the user of recommended activities received from the server. Notifications are customized according to the user's preferences, allowing for options such as push notifications or email. The user can review the notification and either add the suggested activity to their schedule or decline the suggestion.

[0593] When a user accepts or rejects an activity, the device sends this response as feedback to the server. The server uses this feedback to adjust its suggestion algorithm and improve the accuracy of future suggestions.

[0594] As a concrete example, consider a scenario where a user has an hour of free time during their lunch break. The server detects this free time and suggests an activity: visiting a nearby gallery that the user has previously expressed interest in on social media. The device notifies the user of this suggestion, and if the user accepts, the activity is automatically added to their schedule.

[0595] In this way, the present invention efficiently proposes activities that match the user's interests and needs within time constraints, thereby supporting a fulfilling lifestyle for the user.

[0596] The following describes the processing flow.

[0597] Step 1:

[0598] The server accesses the user's schedule database to retrieve the user's appointments. This allows it to identify available time slots and generate free time data.

[0599] Step 2:

[0600] The server collects data related to the user's past posts and interests through SNS APIs. Furthermore, it uses machine learning models to analyze activities and themes that the user is likely to be interested in.

[0601] Step 3:

[0602] The server combines free time data with user interest data to select the most suitable activity. The selection is optimized to suggest beneficial activities within the user's accessible range, taking into account their current location.

[0603] Step 4:

[0604] The device receives suggested activities from the server and displays a notification to the user. The notification format is determined according to the user's settings, and may include push notifications or email.

[0605] Step 5:

[0606] The user reviews the device notification and chooses to accept or reject the suggested activity. The user's choice is recorded by the device.

[0607] Step 6:

[0608] The terminal sends the user's selection results to the server. The server analyzes the results and adjusts the suggestion algorithm. By adjusting the algorithm, the accuracy of future suggestions is improved, and the user's preferences are reflected more accurately.

[0609] (Example 1)

[0610] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0611] In today's information-driven society, individuals have limited time to dedicate to enriching activities based on their hobbies and interests amidst their busy daily lives. Therefore, there is a need for a system that efficiently suggests activities suited to individual free time, thereby improving users' quality of life. However, conventional systems often fail to adequately reflect user interests in their suggestions, and improving the accuracy of real-time suggestions is challenging.

[0612] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0613] In this invention, the server includes means for acquiring the user's time information and identifying spatial availability, means for analyzing the user's interests and past behavioral history, and means for selecting activities that will provide high satisfaction to the user using a generating AI. This makes it possible to suggest activities that are customized to each individual user and are highly accurate and delivered in real time.

[0614] "User time information" refers to information about an individual's available time within a specific period, such as their schedule or appointments.

[0615] "Spatial leeway" refers to unscheduled free time within a user's schedule, providing a gap in time for engaging in new activities.

[0616] "Interest" refers to the matters that a user shows interest in or attention to regarding a particular theme or field.

[0617] "Past behavioral history" refers to a record of various activities and choices a user has made in the past, and serves as data for analyzing their interests and tendencies.

[0618] "Generative AI" refers to a model or method that uses artificial intelligence technology to automatically generate new data, information, or suggestions.

[0619] A "high-satisfaction activity" is an activity that users find interesting and that is considered to improve their quality of life.

[0620] "Information and communication technology" is a general term for technologies and methods for efficiently transmitting, receiving, and processing information.

[0621] "Digital data" refers to information in a format handled by computers and digital devices, which is generated, stored, and transmitted electronically.

[0622] An "activity suggestion algorithm" is a set of calculation procedures and rules used to select activities that are suitable for the user.

[0623] "Location information" is data that indicates a user's geographical location at a specific time, and is used to optimize activities.

[0624] In order to implement this invention, the server, terminal, and user each need to play a specific role.

[0625] The server uses an API from a cloud-based scheduling tool to retrieve the user's schedule information. This tool retrieves the user's schedule information as digital data and stores it in JSON format. Specifically, it is expected that scheduling tools such as the Google Calendar API will be used. The server processes this information to identify the user's time information.

[0626] Next, the server uses social media APIs to analyze the user's interests and past behavior history. For example, it retrieves user "likes" and follow information through the Twitter and Instagram APIs and analyzes it using the Python pandas library. This process identifies the user's areas of interest, and the results are used by the generative AI to select activities that will provide high satisfaction. The generative AI model used is one designed to generate activities based on the user's behavior.

[0627] An example of a prompt for the generating AI model is: "Consider the user's schedule and social media data, and suggest the most suitable activity for their next free time. Example: 1-hour lunch break, interest: art."

[0628] The device serves to notify the user of suggested activities. It uses push notification services such as Firebase Cloud Messaging to communicate selected activities to the user using information and communication technology. These notifications can be customized to the user's preferences, and email notifications are also an option.

[0629] Users can check notifications from their devices and choose to accept or reject suggested activities. This response is sent back to the server and recorded as digital data. Using this data, the server can adaptively improve the activity suggestion algorithm and increase the accuracy of future suggestions.

[0630] In this way, users, servers, and terminals work together to dynamically suggest activities that are optimal for each individual user, thereby improving the user's quality of life.

[0631] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0632] Step 1:

[0633] The server retrieves the user's schedule information. The input is the user's schedule data obtained from the API of a cloud-based scheduling tool. The server analyzes this data, uses a Python library to identify spatial availability, and identifies the next available time slots. The output is a list of available time slots. Specifically, the server requests schedule information using the API key and analyzes the returned data.

[0634] Step 2:

[0635] The server collects and analyzes user interests and past behavioral history. The input is user "likes" and follow information obtained through SNS APIs. The server uses the pandas library to organize the data and extract specific areas of interest. The output is a list of keywords indicating the user's interests. Specifically, the server sends API requests to SNS accounts to collect interest data.

[0636] Step 3:

[0637] The server uses a generative AI model to select activities that will provide high satisfaction to the user. The input consists of a list of free time slots and a list of keywords of the user's interests. The server inputs these as prompts into the generative AI model to generate the optimal activity. The output is the recommended activity. Specifically, the server inputs prompts into the generative AI and analyzes the results obtained.

[0638] Step 4:

[0639] The device notifies the user of recommended activities received from the server. The input is the recommended activities sent from the server. The device uses this information to send a push notification to the user. The output is the notification displayed on the user's device. Specifically, the device uses Firebase Cloud Messaging to send the notification.

[0640] Step 5:

[0641] The user checks the notification from their device and accepts or rejects the offered activity. The input is the notification from the device. The user's choice is recorded as an action and returned to the server as feedback. The output is the result of the user's choice. In concrete terms, the user inputs their choice through the interface on their device.

[0642] Step 6:

[0643] The server receives user selections as feedback and updates the activity suggestion algorithm. The input is the user's selection results. The server analyzes this data and adjusts the algorithm's weights. The resulting output is an updated algorithm designed to improve the accuracy of future suggestions. Specifically, the server saves the feedback to a database and retrains the algorithm.

[0644] (Application Example 1)

[0645] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0646] In modern times, many people find it difficult to manage their time in their daily lives, and there is a particular need to make efficient use of their free time. However, there are limited systems that can suggest meaningful activities tailored to individual lifestyles and interests, and these have not been able to increase user satisfaction. Furthermore, there is a need for suggestions using visual means that can provide more intuitive and immediate information.

[0647] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0648] In this invention, the server includes means for detecting the user's free time, means for extracting the user's usage history and interests, and means for presenting real-time activity information based on spatial circumstances via a visual information display device. This enables efficient and satisfying use of time by suggesting optimized activities to the user and notifying them of these suggestions visually and intuitively.

[0649] "Means for detecting a user's free time" refers to a function that analyzes a user's schedule information to identify time periods when normal activities are not scheduled.

[0650] "Means for extracting user usage history and interests" refers to functions that analyze past behavioral data and records of electronic information exchange to understand the topics of interest and behavioral trends of users.

[0651] "A means of suggesting highly satisfying activities to users" refers to a function that selects the most suitable activities based on the user's free time and interests, and makes recommendations to increase satisfaction.

[0652] "Means of notifying users of suggestions" refers to a means of informing users of selected activities, which is a function that displays or notifies users of information on their devices using visual or auditory methods.

[0653] "Means for collecting user responses and updating the suggestion model" refers to a function that accumulates user responses to suggestions and continuously improves the activity suggestion algorithm based on that data.

[0654] "Means for presenting real-time activity information based on spatial circumstances via a visual information display device" refers to a function that visually displays real-time location information and activities corresponding to the surrounding environment to the user, thereby concretizing the information.

[0655] This system is designed to help users make effective use of their free time and support a more fulfilling lifestyle. The server retrieves the user's schedule information and identifies their free time. This process involves analyzing schedule information stored in a cloud-based database management system. Next, the server uses machine learning models to analyze SNS data and past electronic information exchange data to extract the user's usage history and interests.

[0656] The device is a visual information display device, such as smart glasses, that notifies the user of optimal activity suggestions received from a server. These suggestions include real-time recommendations that take spatial context into account, based on the user's current location. For example, while a user is walking in the city, information about events taking place in a nearby park might be displayed. This intuitive display allows the user to consciously choose their activity.

[0657] Furthermore, user feedback is constantly fed back from the device to the server, contributing to the improvement of the proposed algorithm's accuracy. This feedback is used to train the generative AI model, which then adjusts subsequent proposals to better match user needs.

[0658] For example, if a user is at a cafe during their lunch break on a weekday, the server will recommend beneficial activities in the vicinity based on their location. An example of a prompt to input into the generating AI model would be, "Based on the user's schedule and social media data, please suggest activities that are relevant to their current location." Through this entire process, it is possible to enrich the user's life.

[0659] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0660] Step 1:

[0661] The server retrieves user schedule data from a cloud database. The input is the user's schedule information, which is used to identify free time. Specifically, it uses a database management system to access and analyze the user's calendar data to detect periods of free time.

[0662] Step 2:

[0663] The server acquires SNS data and past electronic information history, and uses a generative AI model to extract user interests. The input is SNS data and usage history, and the output is the user's areas of interest. Specifically, it analyzes text data using natural language processing techniques and lists the user's areas of interest.

[0664] Step 3:

[0665] Based on the above information, the server selects suitable activities and generates activity information optimized in real time based on the user's current location. Inputs include free time, interests, and location information, while output is activity suggestions. The server uses location services to search, select, and recommend events and activities relevant to the user's location.

[0666] Step 4:

[0667] The smart glasses, acting as the terminal, receive suggestions from the server and present them to the user as visual notifications. The input is activity suggestions from the server, and the output is a visual display. Specific actions include displaying information on the screen in a conspicuous manner through the user interface.

[0668] Step 5:

[0669] The user reviews the suggested activity and responds by selecting or rejecting it. The input is a visual notification, and the output is the user's choice. The user looks at the displayed suggestions, makes a selection based on their interests, and sends feedback to the server via the terminal.

[0670] Step 6:

[0671] The server receives user feedback and updates the proposed algorithm. The input is the user's selection and feedback, and the output is the updated proposed model. Specifically, the feedback is input into the machine learning algorithm to tune the model and improve the accuracy of future proposals.

[0672] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0673] The present invention is an activity suggestion system that incorporates an emotion engine to efficiently utilize users' free time and improve user satisfaction. This system recognizes the user's emotional state in real time and adjusts the suggested activities based on that information.

[0674] The server retrieves the user's schedule information and identifies their free time. Next, the server analyzes the user's social media data and past usage history to extract their interests. During this process, the emotion engine analyzes the user's emotional state. Emotions are evaluated in real time using text analysis and image recognition technologies. Based on these evaluation results, the server selects activities to suggest.

[0675] As a concrete example, consider a scenario where a user is feeling stressed and needs to be offered relaxing activities. The server detects signs of stress from the user's facial expressions and text information, and then suggests the most suitable relaxation spot based on the user's time and interests. For example, it might recommend a walk in a nearby park or reading in a quiet cafe.

[0676] The device receives suggestions from the server and notifies the user. Notifications are tailored to the user's settings, with priority given to activities such as stress relief. The user checks the device's notifications and accepts the suggestions, which are automatically added to the schedule.

[0677] Furthermore, by recording user responses, the server can update its suggestion algorithm, enabling it to propose more effective activities in the future. This system leverages emotion engine data to provide a dynamic and personalized experience tailored to the user's psychological state.

[0678] The following describes the processing flow.

[0679] Step 1:

[0680] The server retrieves user schedule data from the database. This identifies the user's current appointments and free time, and prepares the data for analysis.

[0681] Step 2:

[0682] The server collects users' past posts and activity history through APIs from social media and other platforms. The collected data is then analyzed using text to extract user interests and preferences.

[0683] Step 3:

[0684] The emotion engine identifies emotions from collected content to determine the user's situation. It uses keywords in text, as well as data obtained from audio and images, to assess the current emotional state.

[0685] Step 4:

[0686] Based on the interests and emotions identified by the server, the most suitable activities are selected. For example, for a user experiencing stress, activities that contribute to relaxation are prioritized.

[0687] Step 5:

[0688] The device receives recommended activities from the server and displays them to the user as notifications. These notifications are presented prominently within the user interface and designed to attract the user's attention.

[0689] Step 6:

[0690] Users review notifications and accept or reject suggested activities. If accepted, the activity is automatically added to the user's schedule, reducing the effort required for discovery.

[0691] Step 7:

[0692] The device sends the user's selection as feedback to the server. This feedback is used to refine the algorithm and improve the accuracy of future suggestions.

[0693] This process enables personalized activity suggestions that are tailored to the user's emotions and preferences.

[0694] (Example 2)

[0695] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0696] In modern society, many people are unaware of their own emotional state due to the busyness and stress of daily life, and as a result, are unable to effectively utilize their free time. There is a need to improve user satisfaction and reduce stress by suggesting optimal activities based on the user's emotional state and interests.

[0697] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0698] In this invention, the server includes means for detecting the user's free time, means for analyzing the user's emotional state in real time, and means for extracting the user's interests based on the information. This makes it possible for the user to make the most of their free time in the most optimal way.

[0699] "Free time" refers to periods in a user's schedule when no specific appointments are scheduled.

[0700] "Emotional state" refers to information that indicates the user's current psychological or emotional condition, and is derived from text and images.

[0701] "Real-time analysis" refers to processing user data instantly without any time delay and obtaining results immediately.

[0702] "Interest extraction" refers to identifying themes and activities that a user is interested in, based on their past behavior and digital information.

[0703] "Activity suggestions" refer to recommending specific actions or activities to users based on their emotional state and interests.

[0704] "Notifications" refer to sending messages or alerts to inform users about suggested activities.

[0705] "Response recording" refers to saving data on the actions and choices a user makes in response to suggested activities.

[0706] "Updating the suggestion algorithm" means improving the algorithm based on recorded user feedback to enhance the accuracy of future suggestions.

[0707] The present invention is a system that uses multiple computer resources to provide activity suggestions that take into account the user's emotional state. The server first obtains the user's schedule information using a specific API and identifies available time. In this process, data is collected, for example, from a calendar application.

[0708] Next, the server uses natural language processing tools (e.g., spaCy and NLTK) to analyze the user's social media data and past usage history, extracting the user's interests. It also utilizes the Sentiment Analysis API and image recognition technologies (e.g., OpenCV and Amazon Rekognition) as sentiment engines to evaluate the user's emotional state in real time from their text information and photos. This evaluation provides the foundational data needed to guide appropriate activities.

[0709] The server utilizes the user's current location information and leverages geographic information services (e.g., Google Maps API) to optimize activity suggestions. This allows it to select activities available near the user and suggest relaxing spots and entertainment options tailored to the user's emotional state.

[0710] The device is responsible for notifying the user of suggestions from the server. Notifications are delivered via push notifications or email according to the user's preferences, and features such as prioritizing activities that take emotional state into consideration are implemented. When the user accepts a suggestion, it is automatically added to the schedule.

[0711] For example, if the server determines from a user's social media posts or images that they are "feeling stressed," it can suggest a walk in a nearby park or a quiet cafe. The AI ​​model generates prompts like the following to suggest activities that can help the user relax: "Please suggest activities that can help the user relax when they are feeling stressed."

[0712] User responses are recorded, and this data is analyzed by the server and used to update the proposed algorithm. This feedback allows the system to provide optimal, personalized activities for the user over the long term.

[0713] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0714] Step 1:

[0715] The server retrieves the user's schedule information. The input is data from the user's calendar application. The server analyzes this information to identify free time slots that are not scheduled. The output is a list of the user's next free time slots. Specifically, the server retrieves data from the calendar application via an API and determines whether there are appointments or not for each date and time.

[0716] Step 2:

[0717] The server analyzes the user's social media data and past usage history to extract their interests. The input consists of the user's past web browsing history and social media posts. The server processes this data using natural language processing tools to extract specific keywords and topics. The output is a list of keywords indicating the user's interests. Specifically, the server uses a text analysis engine to evaluate frequently occurring words and contexts to identify areas of interest.

[0718] Step 3:

[0719] The server evaluates the user's real-time emotional state. The input is the text and images from the user's latest social media posts. The server uses an emotion engine to analyze the sentiment of the text and images and determine the user's current emotional state. The output is a positive, negative, or neutral emotional state. Specifically, the server uses an emotion analysis API to aggregate positive and negative scores for words, and also uses image recognition to infer emotions from facial expressions.

[0720] Step 4:

[0721] The server selects appropriate activities based on the user's free time, interests, and emotional state. The inputs are free time, user interests, and emotional state. The server uses a generative AI model to generate the optimal activity from this data. The output is the activity suggested to the user. Specifically, the server dynamically generates different activity candidates, evaluates them, and selects the most suitable one.

[0722] Step 5:

[0723] The device notifies the user of selected activity suggestions. The input is the activity suggestion sent from the server. The device sends a push notification to deliver this information to the user. The output is the activity suggestion message displayed on the user's device. Specifically, the device activates the notification system and delivers a message to the user accompanied by an alarm or vibration depending on the importance level.

[0724] Step 6:

[0725] The user reacts to the suggested activity, and the result is recorded. The input is the user's reaction, i.e., their acceptance or rejection of the suggestion. The server stores this information in a reaction log and uses it to improve the suggestion algorithm. The output is the updated algorithm. Specifically, the server periodically takes the user's selection history as training data and forms a feedback loop to improve the accuracy of suggestions in the future.

[0726] (Application Example 2)

[0727] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0728] In modern society, many users experience stress and fatigue in their daily lives. Furthermore, it is difficult to suggest optimal activities based on users' emotional states and interests amidst their busy schedules. Therefore, this invention aims to improve user satisfaction by efficiently utilizing users' free time and suggesting activities that match their emotional state.

[0729] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0730] In this invention, the server includes means for detecting the user's free time, means for extracting the user's usage history and interests, and means for analyzing the user's emotional state using an emotion recognition engine and adjusting the suggested activities. This makes it possible to suggest appropriate activities to the user in real time according to their emotional state.

[0731] "User free time" refers to the period of time when a user has no scheduled appointments and is free to use as they please.

[0732] "Usage history" refers to data that records information about activities and services a user has used in the past.

[0733] "Interest" refers to information related to the subjects or actions that users are interested in.

[0734] An "emotion recognition engine" refers to a program or system that analyzes a user's emotional state from their facial expressions, voice, and text data.

[0735] "Real-time evaluation" refers to a process that analyzes and outputs results instantly at the moment data is generated.

[0736] "Activities" refer to actions, tasks, or activities that become part of a user's life, such as relaxation.

[0737] "Notifications" refer to sending messages or alerts to inform users of information or suggestions.

[0738] "Response" refers to the user's reply or action in response to a suggestion or notification.

[0739] A "proposal model" refers to an algorithm or mechanism for selecting content to propose to the user.

[0740] In this embodiment of the invention, the system primarily consists of a server and terminals. The server has the capability to detect users' free time and process large amounts of data to extract usage history and interests. This includes cloud-based data storage and an analysis engine. The server also analyzes the user's real-time emotional state using an emotion recognition engine. To this end, it uses OpenCV for image recognition technology and a virtual emotion recognition library for emotion analysis.

[0741] The device functions as a user interface, notifying the user of activity suggestions. These notifications can be customized according to user settings; for example, suggestions aimed at stress relief can be prioritized. The device functions as a smartphone or consumer robot, recording user responses and sending feedback to a server.

[0742] This feedback loop allows the suggestion model to be continuously updated and evolve to recommend even more personalized activities to the user. For example, if the device determines that a user is stressed upon returning home, it may play soothing music to create a relaxing environment.

[0743] An example of a prompt suitable for a generative AI model is: "Analyze the user's facial expressions from the image captured by the camera, and if stress is detected, suggest a relaxing activity."

[0744] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0745] Step 1:

[0746] The server retrieves user schedule information and identifies available time slots. The input is the user's schedule data, and the output is the identified available time slots. This process fetches schedules from a database and extracts available time slots using an algorithm.

[0747] Step 2:

[0748] The server analyzes the user's social media data and past usage history to extract their interests. Input is social media posts and past activity logs, and output is information about the user's interests. Data analysis tools are used, and text mining techniques are employed to identify areas of interest.

[0749] Step 3:

[0750] The server uses an emotion recognition engine to analyze the user's emotional state in real time. Input is the user's facial image and voice data, and output is the analyzed emotional state. It utilizes OpenCV and a virtual emotion recognition library to scan facial expressions and voice patterns.

[0751] Step 4:

[0752] The server selects appropriate activities based on the user's free time, interests, and emotional state. The input is free time, interests, and emotional information, and the output is suggested activities. An algorithm is used to generate recommended activities by considering multiple factors.

[0753] Step 5:

[0754] The device receives suggestions from the server and notifies the user. The input is suggestion activity information, and the output is the notification to the user. The notification is displayed on the device as a pop-up or voice guidance.

[0755] Step 6:

[0756] The user checks the notification on their device and selects or rejects an activity based on it. The input is the suggested activity, and the output is the user's response. By selecting an option, the user determines their next action.

[0757] Step 7:

[0758] The server records user responses and updates the suggestion model. The input is user response data, and the output is the updated model. This allows future suggestions to be more personalized and the model to better suit the user's preferences.

[0759] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0760] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0761] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0762] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0763] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0764] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0765] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0766] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0767] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0768] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0769] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0770] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0771] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0772] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0773] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0774] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0775] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0776] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0777] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0778] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0779] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0780] The following is further disclosed regarding the embodiments described above.

[0781] (Claim 1)

[0782] A means of detecting a user's free time,

[0783] A means of extracting user usage history and interests,

[0784] A means of suggesting highly satisfying activities to users,

[0785] A means of notifying users of the proposal,

[0786] A means of recording user reactions and updating the proposed model,

[0787] A system that includes this.

[0788] (Claim 2)

[0789] The system according to claim 1, further comprising means for analyzing data related to a user's past electronic information exchanges and estimating the user's interests.

[0790] (Claim 3)

[0791] The system according to claim 1, further comprising means for optimizing suggestions by utilizing the user's current location information.

[0792] "Example 1"

[0793] (Claim 1)

[0794] A means of obtaining the user's time information and identifying spatial leeway,

[0795] A means of analyzing user interests and past behavioral history,

[0796] A method for selecting activities that provide high satisfaction to users using a generation AI,

[0797] A means of communicating selected activities to the user using information and communication technology,

[0798] A means of collecting user responses as digital data and adaptively improving the activity suggestion algorithm,

[0799] A system that includes this.

[0800] (Claim 2)

[0801] The system according to claim 1, further comprising operations for analyzing a user's electronic information exchange history and evaluating the user's interest trends.

[0802] (Claim 3)

[0803] The system according to claim 1, further comprising a process for spatially optimizing activity suggestions using the user's location information.

[0804] "Application Example 1"

[0805] (Claim 1)

[0806] A means of detecting a user's free time,

[0807] A means of extracting user usage history and interests,

[0808] A means of proposing activities that will satisfy users,

[0809] A means of notifying users of the proposal,

[0810] A means of collecting user feedback and updating the proposed model,

[0811] A means for presenting real-time activity information based on spatial circumstances via a visual information display device,

[0812] A system that includes this.

[0813] (Claim 2)

[0814] The system according to claim 1, further comprising means for analyzing materials related to the user's past electronic information exchanges and estimating the user's interests.

[0815] (Claim 3)

[0816] The system according to claim 1, further comprising means for optimizing suggestions by utilizing the user's current location information.

[0817] "Example 2 of combining an emotion engine"

[0818] (Claim 1)

[0819] A means of detecting a user's free time,

[0820] A means of analyzing the user's emotional state in real time,

[0821] A means of extracting user interests based on information,

[0822] A means of suggesting activities to users that correspond to their emotional state,

[0823] A means of notifying users of the proposal,

[0824] A means to record user feedback and update the proposed algorithm,

[0825] A system that includes this.

[0826] (Claim 2)

[0827] The system according to claim 1, further comprising means for analyzing past data related to digital information and communication and estimating user interests.

[0828] (Claim 3)

[0829] The system according to claim 1, further comprising means for optimizing suggestions by utilizing the user's location information.

[0830] "Application example 2 when combining with an emotional engine"

[0831] (Claim 1)

[0832] A means of detecting a user's free time,

[0833] A means of extracting user usage history and interests,

[0834] A means of proposing activities that will satisfy users,

[0835] A means of notifying users of the proposal,

[0836] A means of recording user reactions and updating the proposed model,

[0837] A means of analyzing the user's emotional state using an emotion recognition engine and adjusting the suggested activities,

[0838] A method for analyzing user image information and evaluating emotions in real time,

[0839] A system that includes this.

[0840] (Claim 2)

[0841] The system according to claim 1, further comprising means for analyzing data related to a user's past electronic information exchanges and estimating the user's interests.

[0842] (Claim 3)

[0843] The system according to claim 1, further comprising means for optimizing suggestions by utilizing the user's current location information. [Explanation of symbols]

[0844] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of detecting a user's free time, A means of extracting user usage history and interests, A means of proposing activities that will satisfy users, A means of notifying users of the proposal, A means of collecting user feedback and updating the proposed model, A means for presenting real-time activity information based on spatial circumstances via a visual information display device, A system that includes this.

2. The system according to claim 1, further comprising means for analyzing materials related to the user's past electronic information exchanges and estimating the user's interests.

3. The system according to claim 1, further comprising means for optimizing suggestions by utilizing the user's current location information.

Images