system

Abstract

Provide a system. 【Solution means】 Means for centrally managing schedule information, Means for proposing an action plan based on the preferences of each household member, Means for automatically collecting and organizing image and video data, Means for sending notifications for specific events or occasions, Means for presenting integrated information using a visual display device, Means for proposing activities based on past data of outdoor activities, Means for making reservations and confirmations for outdoor activities, A system including means for displaying videos and images based on a time series.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern families, it is a complicated and time-consuming task to manage the schedules of each member individually, so unified management is required. Also, there is a growing need because there is no time to make an action plan that all family members can enjoy. Furthermore, there are also problems such as difficulty in recording daily events and difficulty in looking back as memories later.

Means for Solving the Problems

[0005] This invention solves these problems by centrally managing schedule information and proposing action plans based on the preferences of household members. Specifically, it provides a means for integrally managing schedule information on a digital platform and includes a means for automatically generating an optimal action plan based on each member's hobbies and interests. Furthermore, it uses a function to automatically collect and organize image and video data, making it easy to record and look back on memories. For specific events and occasions, a means is provided to send notifications in advance, ensuring that important events are followed.

[0006] "Schedule information" refers to date and time data that indicates an individual's or group's appointments or plans, and is used to manage and coordinate them.

[0007] "Means of centralized management" refers to a method or device that integrates and processes multiple pieces of information in one place, enabling more efficient control and utilization.

[0008] "Means of proposing action plans based on members' preferences" refers to a system or algorithm that analyzes users' hobbies and interests and proposes optimal actions and activities.

[0009] "Means for automatically collecting and organizing image and video data" refers to technologies that mechanically collect photographic and video data, classify it according to date, time, and content, and store it.

[0010] "Means of notifying about specific events or occasions" refers to a system for sending alerts or notifications to users in advance about predefined important events or appointments.

[0011] "Means of automating reservations and purchases" refers to a system that automatically performs necessary reservations and product purchase procedures based on a proposed action plan.

[0012] "Means for analyzing consistency and creating optimized plans" refer to processes and techniques for detecting inconsistencies in diverse information and generating the most efficient and practical plans. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0019] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] This invention is a system for efficiently managing the lives of the entire family, and has three main functions: schedule management, activity suggestions, and storage of growth records.

[0035] Schedule management

[0036] The server receives schedule information entered by each household member via their device and manages it centrally. This integrates parents' work schedules, children's school events, and extracurricular activity schedules into a single platform. The server checks for duplicates, generates an optimized schedule, and delivers it to the device. Users can then use this to manage their time efficiently.

[0037] Specific example:

[0038] When users enter their school events or work meetings into their devices, the server centralizes them and delivers a coordinated schedule that avoids conflicts. For example, if a child's sports day clashes with an important parental meeting, the server will suggest rescheduling the meeting.

[0039] Activity suggestions

[0040] The server suggests activities suitable for weekends and holidays based on the family members' past data and entered preferences. These suggestions include events selected to be enjoyable for the whole family, and the server can also assist with reservations and ticket arrangements.

[0041] Specific example:

[0042] On a fine weekend, the server suggests local festivals tailored to the family's interests. If necessary, the server automatically purchases admission tickets online and sends a confirmation message to the device.

[0043] Saving growth records

[0044] The device automatically uploads and organizes photos and videos taken by the user in their daily life to a server. This allows the server to create digital albums in chronological order, making it easy for the user to relive precious family memories.

[0045] Specific example:

[0046] Photos and videos taken by users at their children's birthday parties are categorized by event and automatically saved to the cloud. Later, family members can view these memories as digital albums on their devices.

[0047] Thus, the present invention provides a comprehensive system for facilitating the daily lives of families and supports each family member in efficiently managing their schedule and keeping records.

[0048] The following describes the processing flow.

[0049] Step 1:

[0050] Users input their schedule information through a smartphone or PC application. Here, they can add detailed information about work appointments, school events, extracurricular activities, etc.

[0051] Step 2:

[0052] The terminal encrypts the entered schedule information and sends it to the server in the cloud. This ensures that the data is transferred securely.

[0053] Step 3:

[0054] The server stores the received schedule information in a database, centralizing the schedules of each household member. The integrated data is then formatted and structured.

[0055] Step 4:

[0056] The server uses AI algorithms to detect schedule overlaps and conflicts and generates optimized schedule proposals to resolve them.

[0057] Step 5:

[0058] The server generates a pre-arranged schedule and sends it to each household member's device. The device then displays this schedule in a visually easy-to-understand format.

[0059] Step 6:

[0060] Users can review the schedule proposed by the server via their device and make modifications or approvals as needed.

[0061] Step 7:

[0062] The server suggests weekend and holiday activities based on the user's past activity history and interests. This includes booking events and purchasing tickets.

[0063] Step 8:

[0064] Once the user reviews the suggested activities and selects one they are interested in, the server automatically completes the necessary reservations and arranges the tickets.

[0065] Step 9:

[0066] The user saves photos and videos taken in their daily life to their device. These media files are automatically uploaded to the cloud.

[0067] Step 10:

[0068] The server analyzes the uploaded media data, classifying and organizing it based on date, time, and events. It then creates a digital album from this data.

[0069] Step 11:

[0070] The device provides a digital album that is accessible to all family members. Through this, users can revisit family memories anytime.

[0071] (Example 1)

[0072] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0073] In daily family life, individual schedules often overlap and clash, making it difficult to make the most of free time. Furthermore, planning activities that all family members can enjoy requires considerable effort, and organizing memorable photos and videos is a cumbersome task.

[0074] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0075] In this invention, the server includes means for aggregating and managing schedule information entered by household members, means for proposing a life plan using a generation model based on the past activity history and preferences of household members, and means for automatically receiving and classifying image and video information. This makes it possible to optimize the schedules of each household member and automate the proposal and planning of activities that everyone can enjoy.

[0076] The term "family member" refers to the individual people who belong to the same household.

[0077] "Schedule information" refers to data that shows the planned activities of individuals or groups at a specific date and time.

[0078] A "generative model" refers to a technology that uses algorithms or software to automatically generate new suggestions based on past data and input information.

[0079] "Image and video information" refers to a data set that includes digital files in the form of still images and videos that have been captured.

[0080] An "algorithm" is a logical framework for performing a series of steps or computational processes designed to solve a specific problem.

[0081] "Optimization" refers to the operation or process of adjusting something to achieve the best possible result based on certain criteria.

[0082] "Planning" is the process of organizing multiple schedules and activities and creating a plan for their effective implementation.

[0083] This system is designed to efficiently manage the lives of the entire family. Its main functions include schedule management, activity suggestions, and the storage of growth records, all of which are achieved through the cooperation of a server and terminals.

[0084] The server receives schedule information from household members sent from terminals and operates a database to aggregate and manage it. This database incorporates algorithms to detect schedule overlaps and optimize schedules, enabling centralized schedule management.

[0085] Specifically, users input their schedules, such as school events or work meetings, using their devices. The server integrates this information and optimizes the plan using algorithms as needed. For example, if a child's sports day clashes with an important business meeting, the server automatically suggests rescheduling the meeting to a different date and time.

[0086] Furthermore, the server employs a generative AI model that suggests activities based on the past activity history and preferences of household members. This makes it possible to provide optimal weekend plans tailored to conditions such as weather and duration. For example, it can recommend participation in local events suitable for sunny days and provide support for purchasing tickets online if necessary.

[0087] Furthermore, the device has a function that automatically sends information about images and videos taken by the user to the server. The server organizes the received data by event and stores it in the cloud, allowing the user to easily view it on the device at a later date.

[0088] In addition to the above process, the AI ​​model generates a prompt that reads, "Design a platform where the schedules of the whole family are centralized. Include a feature to prevent conflicts between parents' work and children's school events." This enables centralized data management and the suggestion of appropriate activities.

[0089] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0090] Step 1:

[0091] The user uses a terminal to enter schedule information for each household member. This information includes data on school events and work meetings. The terminal converts this schedule information into a specified format and sends it to the server.

[0092] Step 2:

[0093] The server stores the schedule information received from the terminals in a database and processes the data for centralized management of household members' schedules. Here, an algorithm is applied to detect duplicate schedules, enabling efficient management.

[0094] Step 3:

[0095] The server detects duplicate schedule information and uses a generation AI model to reference each household member's past data and preferences to generate an optimal schedule. This process performs data calculations that automate schedule adjustments and suggestions. The generated schedule is delivered to the user's device.

[0096] Step 4:

[0097] Users can view the schedule proposed by the server from their terminal and make manual modifications as needed. The user's modifications are then sent back to the server and updated in the database.

[0098] Step 5:

[0099] The server uses a generative AI model to create activities based on past activity history and preferences, in order to suggest activities suitable for each household member. Data processing is performed here, incorporating external event information, and the suggestions are then notified to the user's device.

[0100] Step 6:

[0101] The device automatically uploads images and videos taken by the user to the server. The server categorizes this media data by event and saves it to cloud storage. This process involves data calculations for file organization and backup.

[0102] Step 7:

[0103] Users can view digital albums stored in the cloud through their devices and relive family memories. This makes it easy to access digital media.

[0104] (Application Example 1)

[0105] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0106] In modern households, managing each member's schedule, suggesting activities, and organizing growth records is extremely complex. This makes it difficult for the whole family to efficiently manage their lives together. In particular, there is a need for activity planning based on the individual preferences and needs of each family member, as well as the effective presentation of visual information. A system is needed to address these challenges and make daily family life smoother and more enjoyable.

[0107] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0108] In this invention, the server includes means for centrally managing schedule information, means for proposing action plans based on the preferences of each household member, means for automatically collecting and organizing image and video data, means for presenting integrated information using a visual display device, means for proposing activities based on past data of outings, means for booking and confirming outings, and means for displaying videos and images in chronological order. This makes it possible for each member of the household to check and flexibly manage each other's schedules and activities in real time.

[0109] "A means of centrally managing schedule information" refers to a system that integrates and manages all of a family member's schedules on a single platform.

[0110] "A means of proposing an action plan based on the preferences of each household member" refers to a function that analyzes the preferences and past activity data of each household member and proposes an action plan that is appropriate for them.

[0111] "Means for automatically collecting and organizing image and video data" refers to the process of automatically uploading photos and videos taken by household members to the cloud, and then organizing and saving them.

[0112] "Means of notifying about specific events or occasions" refers to a system for notifying household members of important events or deadlines on the calendar as they approach.

[0113] "Means of presenting integrated information using a visual display device" refers to a function that visually displays integrated schedules and notifications using devices such as smart glasses.

[0114] "A means of suggesting activities based on past data of outings" refers to a system that suggests optimal future activities based on the past outing history of household members.

[0115] "Means for booking and confirming outings" refers to a process that automatically books suggested activities and allows household members to confirm the necessary information.

[0116] "Means for displaying video and images in chronological order" refers to a function that organizes captured video and images in chronological order and makes them viewable on a visual device.

[0117] The system of this invention aims to efficiently manage the lives of the entire family and support the schedules and action plans of each household member. The system is centered around a server and functions by exchanging information with terminals such as smartphones and smart glasses.

[0118] The server receives and centrally manages schedule information. Schedule information is aggregated using the Google® Calendar API, integrating the schedules of household members. This allows for automatic checking of overlapping appointments and the delivery of optimized schedules to devices in real time.

[0119] On user terminals, integrated information is presented using visual display devices (e.g., smart glasses), and important events and schedule changes are notified. This allows household members to easily check their schedules and make decisions smoothly.

[0120] Furthermore, the server uses a generative AI model to suggest activities based on the family members' past outing data. For example, by analyzing preferences for places and events visited in the past, it can suggest new activities based on prompts such as, "Based on the family's past holiday activity data, please suggest the best activities for next weekend." The server also handles booking and confirmation of suggested activities, allowing family members to quickly execute their activity plans.

[0121] Furthermore, image and video data captured by the device are automatically collected using image processing libraries such as OpenCV and organized chronologically. The organized data is uploaded to a server and used as a digital album displayed chronologically. This album is extremely convenient for users to look back on family memories in real time.

[0122] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0123] Step 1:

[0124] The server receives schedule information from each household member's smartphone. The input data consists of each household member's schedule, which is integrated on the server using the Google Calendar API. The server then stores the received schedule data in a database.

[0125] Step 2:

[0126] The server analyzes the integrated schedule information and checks for duplicates. The input is the integrated schedule data received in step 1, and the server compares and analyzes the time of each scheduled event. If duplicates are found, the server generates a suggested optimal schedule as instructed by the program and outputs the reconstructed schedule.

[0127] Step 3:

[0128] The server uses a generation AI model to suggest activities based on past outing data of family members. The input is past outing logs. The server utilizes AI to analyze and process family interests and past event data. Based on this analysis, it suggests activities suitable for holidays and weekends and outputs them to the terminal.

[0129] Step 4:

[0130] The server executes the online booking process based on the proposed activity. The input is the details of the proposed activity, and the server processes it via access to the booking site and outputs booking confirmation information. The user receives a confirmation message on their terminal.

[0131] Step 5:

[0132] The system automatically uploads images and video data captured by the user's device to the server. The input is digital image data captured by the device. The server uses image processing libraries such as OpenCV to organize the data and outputs a digital album based on chronological order. This digital album can be viewed and checked by the user at any time.

[0133] Step 6:

[0134] The smart glasses, which act as a visual display device, present integrated information to the user. The input consists of integrated schedule and activity information transmitted from a server. The smart glasses visually display and output this information. The user then reviews the schedule based on the displayed information and makes adjustments as needed.

[0135] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0136] This invention is a system that centrally manages the schedules of each member of a household and has a function to propose action plans that combine an emotional engine. This invention makes it possible to provide suggestions and notifications that are tailored to the preferences and emotional states of each individual member.

[0137] Integration of schedule management and emotional engine

[0138] The server receives schedule information entered by the user from the terminal and stores it centrally in a database. When the user inputs or automatically acquires emotional data in their daily life, the terminal utilizes an emotion engine to analyze the user's emotional state. This emotional data is also stored on the server along with other schedule information. The server then proposes an optimal schedule that takes the emotional state into account and provides the content to the user's terminal.

[0139] Specific example:

[0140] If emotional data indicating that the user is tired is recorded, the server will prioritize suggesting activities that promote relaxation. For example, instead of a scheduled strenuous exercise event, it will schedule a quieter activity such as watching a movie or taking a walk, and notify the device.

[0141] Action plan proposal and emotional feedback

[0142] The emotion engine analyzes the user's emotional state in real time based on configured parameters. The server receives this analysis result and uses it to improve the accuracy of activity recommendations. For example, if the emotion is determined to be positive, a challenging activity can be recommended.

[0143] Specific example:

[0144] On days when users are feeling energetic, the server sends notifications to their devices encouraging them to participate in outdoor activities or sporting events, and automatically arranges event reservations if necessary.

[0145] Notification customization that reflects emotional state

[0146] The server provides a feature that customizes the content and method of notifications, taking into account the user's emotional state. This ensures that important schedules and events are notified in a way that is appropriate to the user's emotions, improving the user experience.

[0147] Specific example:

[0148] If the user is feeling stressed, the server generates a message in a softer tone and sends it to the device. Conversely, if the user is relaxed, a notification is sent that includes more specific and action-oriented instructions.

[0149] Adding emotion to digital albums

[0150] Photos and videos taken by users in their daily lives are uploaded to a server along with emotional data and organized not only chronologically but also based on emotional tags. Based on this information, the server creates digital albums that record memories, allowing users to look back on the changes in their emotions.

[0151] As described above, the present invention is a system for realizing schedule management and action suggestions that take emotional elements into consideration, and can provide users with a more personalized experience.

[0152] The following describes the processing flow.

[0153] Step 1:

[0154] The user inputs their daily schedule information through their device, and simultaneously, the emotion engine records their current emotional state data. This allows for the collection of both scheduled and emotional information.

[0155] Step 2:

[0156] The terminal collects schedule information and sentiment data, which are then sent to the server using a secure protocol. The server stores this data in a database and prepares it for the next processing step.

[0157] Step 3:

[0158] The server centrally organizes the schedule information it receives and integrates it as the schedule for all household members. During this process, it takes emotional data into consideration and adjusts the priority and content of the schedule accordingly.

[0159] Step 4:

[0160] The server generates an optimal action plan based on information from the emotion engine. This plan also reflects individual preferences and activities from an existing database.

[0161] Step 5:

[0162] The server generates an action plan, which is then delivered to each user's terminal in real time. The terminal visualizes this plan and presents it in a format that is easy for the user to understand.

[0163] Step 6:

[0164] The user reviews the proposed action plan on their device and provides feedback if necessary. It's also possible to notify the emotion engine of adjustments based on changes in their emotions.

[0165] Step 7:

[0166] The emotion engine performs more accurate analysis based on updated emotion data and user feedback. The server receives the results and reviews the content and method of notifications again.

[0167] Step 8:

[0168] The server prepares emotionally responsive, customized notifications and sends them to the user's device at the appropriate time. For example, stress reduction suggestions might be sent as gentle messages.

[0169] Step 9:

[0170] Photos and videos that users record in their daily lives are linked to emotional information and uploaded to a server via their devices. The server analyzes this data, tagging it with emotions and organizing it accordingly.

[0171] Step 10:

[0172] Based on the data organized by the server, a digital album is created. This album displays the user's emotional changes and is accessible from their device.

[0173] (Example 2)

[0174] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0175] Managing the schedules of multiple family members in modern households is complex, and proposing activity plans that take into account each member's feelings and preferences is difficult. Furthermore, there is a lack of effective ways to organize daily digital content and optimize notifications based on emotional information. This creates a need for a suitable personal assistant system to improve the quality of daily life.

[0176] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0177] In this invention, the server includes means for centrally managing schedule data, means for analyzing emotional states and proposing activity plans based on them, and means for collecting and organizing captured digital content along with emotional information. This enables users to receive personalized suggestions based on their individual emotions and schedules.

[0178] "Schedule data" refers to all information about activities and events that a user plans in their daily life, including time, place, and content.

[0179] "Means of centralized management" refers to methods and processes for centrally aggregating and efficiently handling information obtained from multiple data sources.

[0180] "Emotional state" refers to information that describes the psychological situation or mood a user is feeling at a particular point in time.

[0181] "Means for analyzing the state" refer to algorithms and software functions used to process collected data and understand its meaning and trends.

[0182] "Means of proposing an activity plan" refers to methods and processes for showing users the optimal actions or options based on acquired data.

[0183] "Digital content" refers to information stored in electronic format, including various media formats such as images, videos, and music.

[0184] "Emotional information" refers to data that indicates a user's psychological state, usually expressed as numbers or text, and forms the basis for analyzing user needs.

[0185] "Methods for optimizing notifications" refer to the most effective methods and processes for determining the timing and content of information sent to users, tailored to individual circumstances.

[0186] This invention is a system that manages the schedules and emotions of each member of the household to provide appropriate activity suggestions and notifications to individual users. The system includes a server, terminals, an emotion engine, and a generative AI model as its main components.

[0187] The server is positioned to centrally manage schedule data and emotional information provided by each user from their terminal. Schedule data details the user's daily schedule, while emotional information typically represents their psychological state, which is analyzed by an emotional engine. This information is stored in a database on the server and used to generate customized activity plans for each user.

[0188] Users use devices such as smartphones and tablets to input their daily schedules and manually record their emotions. These devices are equipped with an emotion engine to analyze the user's emotional state, enabling automated emotion analysis. The latest emotion recognition technology is used to collect emotion data, generating highly accurate emotion labels.

[0189] The data collected from the device is further analyzed and activity suggestions are made using a server-based AI model. The AI ​​model combines schedule data and emotional information to generate an optimal plan tailored to the user. The results are provided as notifications to the user's device, enabling flexible activity suggestions that align with their lifestyle.

[0190] For example, on a day when a user feels tired, the server uses a generated AI model to suggest relaxing activities such as "watching a movie" or "listening to music." If the user prompts on their device with "What activities should I do today?", appropriate suggestions will be presented in response to the question, "What activities can be suggested if the user's mood is 'relaxed'?"

[0191] This system configuration allows users to enjoy a personalized, high-quality experience tailored to their emotions and schedule at any given time.

[0192] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0193] Step 1:

[0194] The user enters schedule information and sentiment data into the device.

[0195] Input: Information about daily schedules and current emotional state

[0196] Processing: The terminal receives data entered by the user, converts schedules into calendar data format, and analyzes emotional data through an emotion engine. This generates emotional labels (e.g., fatigue, happiness).

[0197] Output: Organized schedule data and analyzed sentiment labels

[0198] Step 2:

[0199] The device sends data to the server.

[0200] Input: Organized schedule data and sentiment labels

[0201] Processing: The terminal securely transmits the collected data to the server via the network for centralized management.

[0202] Output: User-specific schedule data and sentiment information stored on the server.

[0203] Step 3:

[0204] The server performs analysis based on stored data to generate activity suggestions.

[0205] Input: Schedule data and sentiment information stored on the server

[0206] Processing: The server uses a generative AI model to analyze data and generate activity suggestions optimized for the user. The generative AI model leverages information learned from the user's past activity history and emotional patterns.

[0207] Output: Generated activity suggestions (e.g., suggestion to watch a movie while relaxing)

[0208] Step 4:

[0209] The server notifies the user's terminal of the proposed results.

[0210] Input: Generated activity proposals

[0211] Processing: The server customizes the suggestions based on the user's emotional state, creates a notification, and sends it to the device.

[0212] Output: Customized activity suggestions displayed on the user's terminal.

[0213] Step 5:

[0214] The user selects an action in response to a suggestion and enters feedback about the activity performed into the device.

[0215] Input: Feedback / reactions regarding activities carried out based on the suggestion

[0216] Processing: The terminal receives user feedback and prepares to send it to the server. This data will be used to improve future suggestions.

[0217] Output: User feedback collected for future proposals

[0218] Step 6:

[0219] The server adjusts the generated AI model based on the feedback, improving the accuracy of the next suggestion.

[0220] Input: User feedback data

[0221] Processing: The server analyzes the feedback and updates the parameters of the generated AI model to improve the accuracy of the next activity suggestion.

[0222] Output: Improved suggestion capabilities through a refined AI model.

[0223] (Application Example 2)

[0224] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0225] It is necessary to optimally manage the diverse schedules of each family member and propose action plans that take into account each member's emotional state. Conventional systems lack sufficient activity suggestions and notification customization that reflect emotional states, making effective schedule management difficult. Therefore, there is a need to achieve flexible and personalized schedule adjustments based on emotions.

[0226] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0227] In this invention, the server includes means for centrally managing schedule information, means for proposing action plans based on the emotional state of each household member, means for organizing image and video data based on emotional data, and means for customizing notification methods according to the emotional state. As a result, users can receive action plan suggestions optimized for their emotional state.

[0228] "Schedule information" refers to data about the activities and plans of each household member, and should be managed centrally.

[0229] "Emotional state" refers to data that indicates the psychological and emotional state of family members, and is an important element when proposing action plans.

[0230] An "action plan" refers to daily activities and schedules proposed that take into account the schedules and emotional states of each family member.

[0231] "Emotional data" refers to measurements and indicators that show the emotional state of household members, and is used for organizing image and video data and customizing notifications.

[0232] "Notification method" refers to the means and format for delivering information and suggestions to household members, and should be adjusted according to the user's emotional state.

[0233] The system for implementing this invention consists of a platform including a consumer robot that monitors the schedule and emotional state of each member of the household. The server aggregates schedule information input from the user and then utilizes an emotion engine to perform emotion analysis. The emotion data is analyzed, and an optimal action plan is proposed based on the results. The proposed action plan is notified to the user through the user's terminal or the robot's display or voice output function.

[0234] The system uses the following hardware: a microphone for voice recognition, a camera to enable user emotion analysis, and a display for showing notifications. The software includes a schedule management application, the Emotion API as an emotion analysis AI, and a notification customization algorithm that generates statements. The server aggregates this data and proposes action plans.

[0235] For example, if the server determines that the user is emotionally fatigued, it will suggest activities to help them relax, such as watching a movie or doing some light stretching. This suggestion is communicated in a gentle tone to reduce the user's burden.

[0236] An example of a prompt message is, "Based on the emotional data of all family members, please suggest activities that the family can enjoy together this weekend. Please be specific, including suggestions for relaxing movies and snacks." In this way, users can receive suggestions optimized for their emotional state, enabling them to live a more fulfilling life.

[0237] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0238] Step 1:

[0239] The terminal receives schedule information from the user and sends it to the server. The input is the user's specific schedule, and the output is aggregated schedule data. This data is stored on the server.

[0240] Step 2:

[0241] The server uses an emotion engine to analyze emotion data sent from the user's device. The input is data indicating the user's emotional state, and the output is the result of the analysis of the emotional state. This reveals which emotion is dominant.

[0242] Step 3:

[0243] The server generates an action plan based on aggregated schedule data and analyzed emotional states. The input is the schedule and the results of the emotional analysis, and the output is the optimal action plan proposed to the user. This plan is adjusted according to the emotional state.

[0244] Step 4:

[0245] The terminal receives an action plan from the server and notifies the user. The input is the action plan sent from the server, and the output is the specific notification content for the user. The notification content is delivered to the user via display or audio.

[0246] Step 5:

[0247] The user reviews the proposed action plan and enters feedback into the terminal. The input is the user's feedback, and the output is data used to improve the accuracy of future proposals. This feedback is sent to the server and used for future proposals.

[0248] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0249] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0250] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0251] [Second Embodiment]

[0252] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0253] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0254] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0255] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0256] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0257] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0258] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0259] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0260] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0261] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0262] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0263] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0264] This invention is a system for efficiently managing the lives of the entire family, and has three main functions: schedule management, activity suggestions, and storage of growth records.

[0265] Schedule management

[0266] The server receives schedule information entered by each household member via their device and manages it centrally. This integrates parents' work schedules, children's school events, and extracurricular activity schedules into a single platform. The server checks for duplicates, generates an optimized schedule, and delivers it to the device. Users can then use this to manage their time efficiently.

[0267] Specific example:

[0268] When users enter their school events or work meetings into their devices, the server centralizes them and delivers a coordinated schedule that avoids conflicts. For example, if a child's sports day clashes with an important parental meeting, the server will suggest rescheduling the meeting.

[0269] Activity suggestions

[0270] The server suggests activities suitable for weekends and holidays based on the family members' past data and entered preferences. These suggestions include events selected to be enjoyable for the whole family, and the server can also assist with reservations and ticket arrangements.

[0271] Specific example:

[0272] On a fine weekend, the server suggests local festivals tailored to the family's interests. If necessary, the server automatically purchases admission tickets online and sends a confirmation message to the device.

[0273] Saving growth records

[0274] The device automatically uploads and organizes photos and videos taken by the user in their daily life to a server. This allows the server to create digital albums in chronological order, making it easy for the user to relive precious family memories.

[0275] Specific example:

[0276] Photos and videos taken by users at their children's birthday parties are categorized by event and automatically saved to the cloud. Later, family members can view these memories as digital albums on their devices.

[0277] Thus, the present invention provides an integrated system to smooth the daily life of a family and assist each family member to efficiently manage schedules and keep records.

[0278] The following describes the process flow.

[0279] Step 1:

[0280] The user inputs their schedule information through an application on a smartphone or PC. Here, detailed information regarding work schedules, school events, hobbies, etc. is added.

[0281] Step 2:

[0282] The terminal encrypts the input schedule information and sends it to the server via the cloud. This ensures the secure transfer of data.

[0283] Step 3:

[0284] The server saves the received schedule information in a database and unifies the schedules of each family member. The integrated data is formatted and structured.

[0285] Step 4:

[0286] The server uses an AI algorithm to detect schedule overlaps and conflicts and generates an optimized schedule plan to resolve them.

[0287] Step 5:

[0288] The server sends the adjusted schedule generated to the terminals of each family member. The terminal displays this in a visually easy-to-understand format.

[0289] Step 6:

[0290] Users can review the schedule proposed by the server via their device and make modifications or approvals as needed.

[0291] Step 7:

[0292] The server suggests weekend and holiday activities based on the user's past activity history and interests. This includes booking events and purchasing tickets.

[0293] Step 8:

[0294] Once the user reviews the suggested activities and selects one they are interested in, the server automatically completes the necessary reservations and arranges the tickets.

[0295] Step 9:

[0296] The user saves photos and videos taken in their daily life to their device. These media files are automatically uploaded to the cloud.

[0297] Step 10:

[0298] The server analyzes the uploaded media data, classifying and organizing it based on date, time, and events. It then creates a digital album from this data.

[0299] Step 11:

[0300] The device provides a digital album that is accessible to all family members. Through this, users can revisit family memories anytime.

[0301] (Example 1)

[0302] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0303] In the daily life of a family, there is often a problem that each individual schedule overlaps and conflicts, making it difficult to optimally utilize free time. Additionally, there is an issue that it takes effort to plan activities that all family members can enjoy, and further, the organization of memorable images and videos is cumbersome.

[0304] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0305] In this invention, the server includes means for aggregating and managing schedule information input by family members, means for proposing a life plan using a generation model based on the past activity history and preferences of family members, and means for automatically receiving and classifying image and video information. Thereby, it becomes possible to optimize the schedules of each family member and automate the proposal and planning of activities that all members can enjoy.

[0306] "Family members" refers to the concept of individuals belonging to the same family.

[0307] "Schedule information" is data indicating individual or group activity schedules at specific dates and times.

[0308] "Generation model" refers to a technology of an algorithm or software that automatically generates new proposals based on past data and input information.

[0309] "Image and video information" refers to a data set including digital files in the form of photographed still images and moving images.

[0310] "Algorithm" is a logical framework for performing a series of procedures or computational processes designed to solve a specific problem.

[0311] "Optimization" refers to an operation or process of adjusting so that the result becomes the best based on a certain criterion.

[0312] "Planning" is the process of organizing multiple schedules and activities and creating a plan for their effective implementation.

[0313] This system is designed to efficiently manage the lives of the entire family. Its main functions include schedule management, activity suggestions, and the storage of growth records, all of which are achieved through the cooperation of a server and terminals.

[0314] The server receives schedule information from household members sent from terminals and operates a database to aggregate and manage it. This database incorporates algorithms to detect schedule overlaps and optimize schedules, enabling centralized schedule management.

[0315] Specifically, users input their schedules, such as school events or work meetings, using their devices. The server integrates this information and optimizes the plan using algorithms as needed. For example, if a child's sports day clashes with an important business meeting, the server automatically suggests rescheduling the meeting to a different date and time.

[0316] Furthermore, the server employs a generative AI model that suggests activities based on the past activity history and preferences of household members. This makes it possible to provide optimal weekend plans tailored to conditions such as weather and duration. For example, it can recommend participation in local events suitable for sunny days and provide support for purchasing tickets online if necessary.

[0317] Furthermore, the device has a function that automatically sends information about images and videos taken by the user to the server. The server organizes the received data by event and stores it in the cloud, allowing the user to easily view it on the device at a later date.

[0318] In addition to the above process, the AI ​​model generates a prompt that reads, "Design a platform where the schedules of the whole family are centralized. Include a feature to prevent conflicts between parents' work and children's school events." This enables centralized data management and the suggestion of appropriate activities.

[0319] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0320] Step 1:

[0321] The user uses a terminal to enter schedule information for each household member. This information includes data on school events and work meetings. The terminal converts this schedule information into a specified format and sends it to the server.

[0322] Step 2:

[0323] The server stores the schedule information received from the terminals in a database and processes the data for centralized management of household members' schedules. Here, an algorithm is applied to detect duplicate schedules, enabling efficient management.

[0324] Step 3:

[0325] The server detects duplicate schedule information and uses a generation AI model to reference each household member's past data and preferences to generate an optimal schedule. This process performs data calculations that automate schedule adjustments and suggestions. The generated schedule is delivered to the user's device.

[0326] Step 4:

[0327] Users can view the schedule proposed by the server from their terminal and make manual modifications as needed. The user's modifications are then sent back to the server and updated in the database.

[0328] Step 5:

[0329] The server uses a generative AI model to create activities based on past activity history and preferences, in order to suggest activities suitable for each household member. Data processing is performed here, incorporating external event information, and the suggestions are then notified to the user's device.

[0330] Step 6:

[0331] The device automatically uploads images and videos taken by the user to the server. The server categorizes this media data by event and saves it to cloud storage. This process involves data calculations for file organization and backup.

[0332] Step 7:

[0333] Users can view digital albums stored in the cloud through their devices and relive family memories. This makes it easy to access digital media.

[0334] (Application Example 1)

[0335] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0336] In modern households, managing each member's schedule, suggesting activities, and organizing growth records is extremely complex. This makes it difficult for the whole family to efficiently manage their lives together. In particular, there is a need for activity planning based on the individual preferences and needs of each family member, as well as the effective presentation of visual information. A system is needed to address these challenges and make daily family life smoother and more enjoyable.

[0337] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0338] In this invention, the server includes means for centrally managing schedule information, means for proposing action plans based on the preferences of each household member, means for automatically collecting and organizing image and video data, means for presenting integrated information using a visual display device, means for proposing activities based on past data of outings, means for booking and confirming outings, and means for displaying videos and images in chronological order. This makes it possible for each member of the household to check and flexibly manage each other's schedules and activities in real time.

[0339] "A means of centrally managing schedule information" refers to a system that integrates and manages all of a family member's schedules on a single platform.

[0340] "A means of proposing an action plan based on the preferences of each household member" refers to a function that analyzes the preferences and past activity data of each household member and proposes an action plan that is appropriate for them.

[0341] "Means for automatically collecting and organizing image and video data" refers to the process of automatically uploading photos and videos taken by household members to the cloud, and then organizing and saving them.

[0342] "Means of notifying about specific events or occasions" refers to a system for notifying household members of important events or deadlines on the calendar as they approach.

[0343] "Means of presenting integrated information using a visual display device" refers to a function that visually displays integrated schedules and notifications using devices such as smart glasses.

[0344] "A means of suggesting activities based on past data of outings" refers to a system that suggests optimal future activities based on the past outing history of household members.

[0345] "Means for booking and confirming outings" refers to a process that automatically books suggested activities and allows household members to confirm the necessary information.

[0346] "Means for displaying video and images in chronological order" refers to a function that organizes captured video and images in chronological order and makes them viewable on a visual device.

[0347] The system of this invention aims to efficiently manage the lives of the entire family and support the schedules and action plans of each household member. The system is centered around a server and functions by exchanging information with terminals such as smartphones and smart glasses.

[0348] The server receives and centrally manages schedule information. Schedule information is aggregated using the Google Calendar API, integrating the schedules of household members. This allows for automatic checking of overlapping appointments and the delivery of optimized schedules to devices in real time.

[0349] On user terminals, integrated information is presented using visual display devices (e.g., smart glasses), and important events and schedule changes are notified. This allows household members to easily check their schedules and make decisions smoothly.

[0350] Furthermore, the server uses a generative AI model to suggest activities based on the family members' past outing data. For example, by analyzing preferences for places and events visited in the past, it can suggest new activities based on prompts such as, "Based on the family's past holiday activity data, please suggest the best activities for next weekend." The server also handles booking and confirmation of suggested activities, allowing family members to quickly execute their activity plans.

[0351] Furthermore, image and video data captured by the device are automatically collected using image processing libraries such as OpenCV and organized chronologically. The organized data is uploaded to a server and used as a digital album displayed chronologically. This album is extremely convenient for users to look back on family memories in real time.

[0352] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0353] Step 1:

[0354] The server receives schedule information from each household member's smartphone. The input data consists of each household member's schedule, which is integrated on the server using the Google Calendar API. The server then stores the received schedule data in a database.

[0355] Step 2:

[0356] The server analyzes the integrated schedule information and checks for duplicates. The input is the integrated schedule data received in step 1, and the server compares and analyzes the time of each scheduled event. If duplicates are found, the server generates a suggested optimal schedule as instructed by the program and outputs the reconstructed schedule.

[0357] Step 3:

[0358] The server uses a generation AI model to suggest activities based on past outing data of family members. The input is past outing logs. The server utilizes AI to analyze and process family interests and past event data. Based on this analysis, it suggests activities suitable for holidays and weekends and outputs them to the terminal.

[0359] Step 4:

[0360] The server executes the online booking process based on the proposed activity. The input is the details of the proposed activity, and the server processes it via access to the booking site and outputs booking confirmation information. The user receives a confirmation message on their terminal.

[0361] Step 5:

[0362] The system automatically uploads images and video data captured by the user's device to the server. The input is digital image data captured by the device. The server uses image processing libraries such as OpenCV to organize the data and outputs a digital album based on chronological order. This digital album can be viewed and checked by the user at any time.

[0363] Step 6:

[0364] The smart glasses, which act as a visual display device, present integrated information to the user. The input consists of integrated schedule and activity information transmitted from a server. The smart glasses visually display and output this information. The user then reviews the schedule based on the displayed information and makes adjustments as needed.

[0365] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0366] This invention is a system that centrally manages the schedules of each member of a household and has a function to propose action plans that combine an emotional engine. This invention makes it possible to provide suggestions and notifications that are tailored to the preferences and emotional states of each individual member.

[0367] Integration of schedule management and emotional engine

[0368] The server receives schedule information entered by the user from the terminal and stores it centrally in a database. When the user inputs or automatically acquires emotional data in their daily life, the terminal utilizes an emotion engine to analyze the user's emotional state. This emotional data is also stored on the server along with other schedule information. The server then proposes an optimal schedule that takes the emotional state into account and provides the content to the user's terminal.

[0369] Specific example:

[0370] If emotional data indicating that the user is tired is recorded, the server will prioritize suggesting activities that promote relaxation. For example, instead of a scheduled strenuous exercise event, it will schedule a quieter activity such as watching a movie or taking a walk, and notify the device.

[0371] Action plan proposal and emotional feedback

[0372] The emotion engine analyzes the user's emotional state in real time based on configured parameters. The server receives this analysis result and uses it to improve the accuracy of activity recommendations. For example, if the emotion is determined to be positive, a challenging activity can be recommended.

[0373] Specific example:

[0374] On days when users are feeling energetic, the server sends notifications to their devices encouraging them to participate in outdoor activities or sporting events, and automatically arranges event reservations if necessary.

[0375] Notification customization that reflects emotional state

[0376] The server provides a feature that customizes the content and method of notifications, taking into account the user's emotional state. This ensures that important schedules and events are notified in a way that is appropriate to the user's emotions, improving the user experience.

[0377] Specific example:

[0378] If the user is feeling stressed, the server generates a message in a softer tone and sends it to the device. Conversely, if the user is relaxed, a notification is sent that includes more specific and action-oriented instructions.

[0379] Adding emotion to digital albums

[0380] Photos and videos taken by users in their daily lives are uploaded to a server along with emotional data and organized not only chronologically but also based on emotional tags. Based on this information, the server creates digital albums that record memories, allowing users to look back on the changes in their emotions.

[0381] As described above, the present invention is a system for realizing schedule management and action suggestions that take emotional elements into consideration, and can provide users with a more personalized experience.

[0382] The following describes the processing flow.

[0383] Step 1:

[0384] The user inputs their daily schedule information through their device, and simultaneously, the emotion engine records their current emotional state data. This allows for the collection of both scheduled and emotional information.

[0385] Step 2:

[0386] The terminal collects schedule information and sentiment data, which are then sent to the server using a secure protocol. The server stores this data in a database and prepares it for the next processing step.

[0387] Step 3:

[0388] The server centrally organizes the schedule information it receives and integrates it as the schedule for all household members. During this process, it takes emotional data into consideration and adjusts the priority and content of the schedule accordingly.

[0389] Step 4:

[0390] The server generates an optimal action plan based on information from the emotion engine. This plan also reflects individual preferences and activities from an existing database.

[0391] Step 5:

[0392] The server generates an action plan, which is then delivered to each user's terminal in real time. The terminal visualizes this plan and presents it in a format that is easy for the user to understand.

[0393] Step 6:

[0394] The user reviews the proposed action plan on their device and provides feedback if necessary. It's also possible to notify the emotion engine of adjustments based on changes in their emotions.

[0395] Step 7:

[0396] The emotion engine performs more accurate analysis based on updated emotion data and user feedback. The server receives the results and reviews the content and method of notifications again.

[0397] Step 8:

[0398] The server prepares emotionally responsive, customized notifications and sends them to the user's device at the appropriate time. For example, stress reduction suggestions might be sent as gentle messages.

[0399] Step 9:

[0400] Photos and videos that users record in their daily lives are linked to emotional information and uploaded to a server via their devices. The server analyzes this data, tagging it with emotions and organizing it accordingly.

[0401] Step 10:

[0402] Based on the data organized by the server, a digital album is created. This album displays the user's emotional changes and is accessible from their device.

[0403] (Example 2)

[0404] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0405] Managing the schedules of multiple family members in modern households is complex, and proposing activity plans that take into account each member's feelings and preferences is difficult. Furthermore, there is a lack of effective ways to organize daily digital content and optimize notifications based on emotional information. This creates a need for a suitable personal assistant system to improve the quality of daily life.

[0406] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0407] In this invention, the server includes means for centrally managing schedule data, means for analyzing emotional states and proposing activity plans based on them, and means for collecting and organizing captured digital content along with emotional information. This enables users to receive personalized suggestions based on their individual emotions and schedules.

[0408] "Schedule data" refers to all information about activities and events that a user plans in their daily life, including time, place, and content.

[0409] "Means of centralized management" refers to methods and processes for centrally aggregating and efficiently handling information obtained from multiple data sources.

[0410] "Emotional state" refers to information that describes the psychological situation or mood a user is feeling at a particular point in time.

[0411] "Means for analyzing the state" refer to algorithms and software functions used to process collected data and understand its meaning and trends.

[0412] "Means of proposing an activity plan" refers to methods and processes for showing users the optimal actions or options based on acquired data.

[0413] "Digital content" refers to information stored in electronic format, including various media formats such as images, videos, and music.

[0414] "Emotional information" refers to data that indicates a user's psychological state, usually expressed as numbers or text, and forms the basis for analyzing user needs.

[0415] "Methods for optimizing notifications" refer to the most effective methods and processes for determining the timing and content of information sent to users, tailored to individual circumstances.

[0416] This invention is a system that manages the schedules and emotions of each member of the household to provide appropriate activity suggestions and notifications to individual users. The system includes a server, terminals, an emotion engine, and a generative AI model as its main components.

[0417] The server is positioned to centrally manage schedule data and emotional information provided by each user from their terminal. Schedule data details the user's daily schedule, while emotional information typically represents their psychological state, which is analyzed by an emotional engine. This information is stored in a database on the server and used to generate customized activity plans for each user.

[0418] Users use devices such as smartphones and tablets to input their daily schedules and manually record their emotions. These devices are equipped with an emotion engine to analyze the user's emotional state, enabling automated emotion analysis. The latest emotion recognition technology is used to collect emotion data, generating highly accurate emotion labels.

[0419] The data collected from the device is further analyzed and activity suggestions are made using a server-based AI model. The AI ​​model combines schedule data and emotional information to generate an optimal plan tailored to the user. The results are provided as notifications to the user's device, enabling flexible activity suggestions that align with their lifestyle.

[0420] For example, on a day when a user feels tired, the server uses a generated AI model to suggest relaxing activities such as "watching a movie" or "listening to music." If the user prompts on their device with "What activities should I do today?", appropriate suggestions will be presented in response to the question, "What activities can be suggested if the user's mood is 'relaxed'?"

[0421] This system configuration allows users to enjoy a personalized, high-quality experience tailored to their emotions and schedule at any given time.

[0422] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0423] Step 1:

[0424] The user enters schedule information and sentiment data into the device.

[0425] Input: Information about daily schedules and current emotional state

[0426] Processing: The terminal receives data entered by the user, converts schedules into calendar data format, and analyzes emotional data through an emotion engine. This generates emotional labels (e.g., fatigue, happiness).

[0427] Output: Organized schedule data and analyzed sentiment labels

[0428] Step 2:

[0429] The device sends data to the server.

[0430] Input: Organized schedule data and sentiment labels

[0431] Processing: The terminal securely transmits the collected data to the server via the network for centralized management.

[0432] Output: User-specific schedule data and sentiment information stored on the server.

[0433] Step 3:

[0434] The server performs analysis based on stored data to generate activity suggestions.

[0435] Input: Schedule data and sentiment information stored on the server

[0436] Processing: The server uses a generative AI model to analyze data and generate activity suggestions optimized for the user. The generative AI model leverages information learned from the user's past activity history and emotional patterns.

[0437] Output: Generated activity suggestions (e.g., suggestion to watch a movie while relaxing)

[0438] Step 4:

[0439] The server notifies the user's terminal of the proposed results.

[0440] Input: Generated activity proposals

[0441] Processing: The server customizes the suggestions based on the user's emotional state, creates a notification, and sends it to the device.

[0442] Output: Customized activity suggestions displayed on the user's terminal.

[0443] Step 5:

[0444] The user selects an action in response to a suggestion and enters feedback about the activity performed into the device.

[0445] Input: Feedback / reactions regarding activities carried out based on the suggestion

[0446] Processing: The terminal receives user feedback and prepares to send it to the server. This data will be used to improve future suggestions.

[0447] Output: User feedback collected for future proposals

[0448] Step 6:

[0449] The server adjusts the generated AI model based on the feedback, improving the accuracy of the next suggestion.

[0450] Input: User feedback data

[0451] Processing: The server analyzes the feedback and updates the parameters of the generated AI model to improve the accuracy of the next activity suggestion.

[0452] Output: Improved suggestion capabilities through a refined AI model.

[0453] (Application Example 2)

[0454] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0455] It is necessary to optimally manage the diverse schedules of each family member and propose action plans that take into account each member's emotional state. Conventional systems lack sufficient activity suggestions and notification customization that reflect emotional states, making effective schedule management difficult. Therefore, there is a need to achieve flexible and personalized schedule adjustments based on emotions.

[0456] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0457] In this invention, the server includes means for centrally managing schedule information, means for proposing action plans based on the emotional state of each household member, means for organizing image and video data based on emotional data, and means for customizing notification methods according to the emotional state. As a result, users can receive action plan suggestions optimized for their emotional state.

[0458] "Schedule information" refers to data about the activities and plans of each household member, and should be managed centrally.

[0459] "Emotional state" refers to data that indicates the psychological and emotional state of family members, and is an important element when proposing action plans.

[0460] An "action plan" refers to daily activities and schedules proposed that take into account the schedules and emotional states of each family member.

[0461] "Emotional data" refers to measurements and indicators that show the emotional state of household members, and is used for organizing image and video data and customizing notifications.

[0462] "Notification method" refers to the means and format for delivering information and suggestions to household members, and should be adjusted according to the user's emotional state.

[0463] The system for implementing this invention consists of a platform including a consumer robot that monitors the schedule and emotional state of each member of the household. The server aggregates schedule information input from the user and then utilizes an emotion engine to perform emotion analysis. The emotion data is analyzed, and an optimal action plan is proposed based on the results. The proposed action plan is notified to the user through the user's terminal or the robot's display or voice output function.

[0464] The system uses the following hardware: a microphone for voice recognition, a camera to enable user emotion analysis, and a display for showing notifications. The software includes a schedule management application, the Emotion API as an emotion analysis AI, and a notification customization algorithm that generates statements. The server aggregates this data and proposes action plans.

[0465] For example, if the server determines that the user is emotionally fatigued, it will suggest activities to help them relax, such as watching a movie or doing some light stretching. This suggestion is communicated in a gentle tone to reduce the user's burden.

[0466] An example of a prompt message is, "Based on the emotional data of all family members, please suggest activities that the family can enjoy together this weekend. Please be specific, including suggestions for relaxing movies and snacks." In this way, users can receive suggestions optimized for their emotional state, enabling them to live a more fulfilling life.

[0467] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0468] Step 1:

[0469] The terminal receives schedule information from the user and sends it to the server. The input is the user's specific schedule, and the output is aggregated schedule data. This data is stored on the server.

[0470] Step 2:

[0471] The server uses an emotion engine to analyze emotion data sent from the user's device. The input is data indicating the user's emotional state, and the output is the result of the analysis of the emotional state. This reveals which emotion is dominant.

[0472] Step 3:

[0473] The server generates an action plan based on aggregated schedule data and analyzed emotional states. The input is the schedule and the results of the emotional analysis, and the output is the optimal action plan proposed to the user. This plan is adjusted according to the emotional state.

[0474] Step 4:

[0475] The terminal receives an action plan from the server and notifies the user. The input is the action plan sent from the server, and the output is the specific notification content for the user. The notification content is delivered to the user via display or audio.

[0476] Step 5:

[0477] The user reviews the proposed action plan and enters feedback into the terminal. The input is the user's feedback, and the output is data used to improve the accuracy of future proposals. This feedback is sent to the server and used for future proposals.

[0478] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0479] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0480] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0481] [Third Embodiment]

[0482] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0483] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0484] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0485] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0486] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0487] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0488] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0489] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0490] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0491] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0492] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0493] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0494] This invention is a system for efficiently managing the lives of the entire family, and has three main functions: schedule management, activity suggestions, and storage of growth records.

[0495] Schedule management

[0496] The server receives schedule information entered by each household member via their device and manages it centrally. This integrates parents' work schedules, children's school events, and extracurricular activity schedules into a single platform. The server checks for duplicates, generates an optimized schedule, and delivers it to the device. Users can then use this to manage their time efficiently.

[0497] Specific example:

[0498] When users enter their school events or work meetings into their devices, the server centralizes them and delivers a coordinated schedule that avoids conflicts. For example, if a child's sports day clashes with an important parental meeting, the server will suggest rescheduling the meeting.

[0499] Activity suggestions

[0500] The server suggests activities suitable for weekends and holidays based on the family members' past data and entered preferences. These suggestions include events selected to be enjoyable for the whole family, and the server can also assist with reservations and ticket arrangements.

[0501] Specific example:

[0502] On a fine weekend, the server suggests local festivals tailored to the family's interests. If necessary, the server automatically purchases admission tickets online and sends a confirmation message to the device.

[0503] Saving growth records

[0504] The device automatically uploads and organizes photos and videos taken by the user in their daily life to a server. This allows the server to create digital albums in chronological order, making it easy for the user to relive precious family memories.

[0505] Specific example:

[0506] Photos and videos taken by users at their children's birthday parties are categorized by event and automatically saved to the cloud. Later, family members can view these memories as digital albums on their devices.

[0507] Thus, the present invention provides a comprehensive system for facilitating the daily lives of families and supports each family member in efficiently managing their schedule and keeping records.

[0508] The following describes the processing flow.

[0509] Step 1:

[0510] Users input their schedule information through a smartphone or PC application. Here, they can add detailed information about work appointments, school events, extracurricular activities, etc.

[0511] Step 2:

[0512] The terminal encrypts the entered schedule information and sends it to the server in the cloud. This ensures that the data is transferred securely.

[0513] Step 3:

[0514] The server stores the received schedule information in a database, centralizing the schedules of each household member. The integrated data is then formatted and structured.

[0515] Step 4:

[0516] The server uses AI algorithms to detect schedule overlaps and conflicts and generates optimized schedule proposals to resolve them.

[0517] Step 5:

[0518] The server generates a pre-arranged schedule and sends it to each household member's device. The device then displays this schedule in a visually easy-to-understand format.

[0519] Step 6:

[0520] Users can review the schedule proposed by the server via their device and make modifications or approvals as needed.

[0521] Step 7:

[0522] The server suggests weekend and holiday activities based on the user's past activity history and interests. This includes booking events and purchasing tickets.

[0523] Step 8:

[0524] Once the user reviews the suggested activities and selects one they are interested in, the server automatically completes the necessary reservations and arranges the tickets.

[0525] Step 9:

[0526] The user saves photos and videos taken in their daily life to their device. These media files are automatically uploaded to the cloud.

[0527] Step 10:

[0528] The server analyzes the uploaded media data, classifying and organizing it based on date, time, and events. It then creates a digital album from this data.

[0529] Step 11:

[0530] The device provides a digital album that is accessible to all family members. Through this, users can revisit family memories anytime.

[0531] (Example 1)

[0532] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0533] In daily family life, individual schedules often overlap and clash, making it difficult to make the most of free time. Furthermore, planning activities that all family members can enjoy requires considerable effort, and organizing memorable photos and videos is a cumbersome task.

[0534] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0535] In this invention, the server includes means for aggregating and managing schedule information entered by household members, means for proposing a life plan using a generation model based on the past activity history and preferences of household members, and means for automatically receiving and classifying image and video information. This makes it possible to optimize the schedules of each household member and automate the proposal and planning of activities that everyone can enjoy.

[0536] The term "family member" refers to the individual people who belong to the same household.

[0537] "Schedule information" refers to data that shows the planned activities of individuals or groups at a specific date and time.

[0538] A "generative model" refers to a technology that uses algorithms or software to automatically generate new suggestions based on past data and input information.

[0539] "Image and video information" refers to a data set that includes digital files in the form of still images and videos that have been captured.

[0540] An "algorithm" is a logical framework for performing a series of steps or computational processes designed to solve a specific problem.

[0541] "Optimization" refers to the operation or process of adjusting something to achieve the best possible result based on certain criteria.

[0542] "Planning" is the process of organizing multiple schedules and activities and creating a plan for their effective implementation.

[0543] This system is designed to efficiently manage the lives of the entire family. Its main functions include schedule management, activity suggestions, and the storage of growth records, all of which are achieved through the cooperation of a server and terminals.

[0544] The server receives schedule information from household members sent from terminals and operates a database to aggregate and manage it. This database incorporates algorithms to detect schedule overlaps and optimize schedules, enabling centralized schedule management.

[0545] Specifically, users input their schedules, such as school events or work meetings, using their devices. The server integrates this information and optimizes the plan using algorithms as needed. For example, if a child's sports day clashes with an important business meeting, the server automatically suggests rescheduling the meeting to a different date and time.

[0546] Furthermore, the server employs a generative AI model that suggests activities based on the past activity history and preferences of household members. This makes it possible to provide optimal weekend plans tailored to conditions such as weather and duration. For example, it can recommend participation in local events suitable for sunny days and provide support for purchasing tickets online if necessary.

[0547] Furthermore, the device has a function that automatically sends information about images and videos taken by the user to the server. The server organizes the received data by event and stores it in the cloud, allowing the user to easily view it on the device at a later date.

[0548] In addition to the above process, the AI ​​model generates a prompt that reads, "Design a platform where the schedules of the whole family are centralized. Include a feature to prevent conflicts between parents' work and children's school events." This enables centralized data management and the suggestion of appropriate activities.

[0549] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0550] Step 1:

[0551] The user uses a terminal to enter schedule information for each household member. This information includes data on school events and work meetings. The terminal converts this schedule information into a specified format and sends it to the server.

[0552] Step 2:

[0553] The server stores the schedule information received from the terminals in a database and processes the data for centralized management of household members' schedules. Here, an algorithm is applied to detect duplicate schedules, enabling efficient management.

[0554] Step 3:

[0555] The server detects duplicate schedule information and uses a generation AI model to reference each household member's past data and preferences to generate an optimal schedule. This process performs data calculations that automate schedule adjustments and suggestions. The generated schedule is delivered to the user's device.

[0556] Step 4:

[0557] Users can view the schedule proposed by the server from their terminal and make manual modifications as needed. The user's modifications are then sent back to the server and updated in the database.

[0558] Step 5:

[0559] The server uses a generative AI model to create activities based on past activity history and preferences, in order to suggest activities suitable for each household member. Data processing is performed here, incorporating external event information, and the suggestions are then notified to the user's device.

[0560] Step 6:

[0561] The device automatically uploads images and videos taken by the user to the server. The server categorizes this media data by event and saves it to cloud storage. This process involves data calculations for file organization and backup.

[0562] Step 7:

[0563] Users can view digital albums stored in the cloud through their devices and relive family memories. This makes it easy to access digital media.

[0564] (Application Example 1)

[0565] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0566] In modern households, managing each member's schedule, suggesting activities, and organizing growth records is extremely complex. This makes it difficult for the whole family to efficiently manage their lives together. In particular, there is a need for activity planning based on the individual preferences and needs of each family member, as well as the effective presentation of visual information. A system is needed to address these challenges and make daily family life smoother and more enjoyable.

[0567] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0568] In this invention, the server includes means for centrally managing schedule information, means for proposing action plans based on the preferences of each household member, means for automatically collecting and organizing image and video data, means for presenting integrated information using a visual display device, means for proposing activities based on past data of outings, means for booking and confirming outings, and means for displaying videos and images in chronological order. This makes it possible for each member of the household to check and flexibly manage each other's schedules and activities in real time.

[0569] "A means of centrally managing schedule information" refers to a system that integrates and manages all of a family member's schedules on a single platform.

[0570] "A means of proposing an action plan based on the preferences of each household member" refers to a function that analyzes the preferences and past activity data of each household member and proposes an action plan that is appropriate for them.

[0571] "Means for automatically collecting and organizing image and video data" refers to the process of automatically uploading photos and videos taken by household members to the cloud, and then organizing and saving them.

[0572] "Means of notifying about specific events or occasions" refers to a system for notifying household members of important events or deadlines on the calendar as they approach.

[0573] "Means of presenting integrated information using a visual display device" refers to a function that visually displays integrated schedules and notifications using devices such as smart glasses.

[0574] "A means of suggesting activities based on past data of outings" refers to a system that suggests optimal future activities based on the past outing history of household members.

[0575] "Means for booking and confirming outings" refers to a process that automatically books suggested activities and allows household members to confirm the necessary information.

[0576] "Means for displaying video and images in chronological order" refers to a function that organizes captured video and images in chronological order and makes them viewable on a visual device.

[0577] The system of this invention aims to efficiently manage the lives of the entire family and support the schedules and action plans of each household member. The system is centered around a server and functions by exchanging information with terminals such as smartphones and smart glasses.

[0578] The server receives and centrally manages schedule information. Schedule information is aggregated using the Google Calendar API, integrating the schedules of household members. This allows for automatic checking of overlapping appointments and the delivery of optimized schedules to devices in real time.

[0579] On user terminals, integrated information is presented using visual display devices (e.g., smart glasses), and important events and schedule changes are notified. This allows household members to easily check their schedules and make decisions smoothly.

[0580] Furthermore, the server uses a generative AI model to suggest activities based on the family members' past outing data. For example, by analyzing preferences for places and events visited in the past, it can suggest new activities based on prompts such as, "Based on the family's past holiday activity data, please suggest the best activities for next weekend." The server also handles booking and confirmation of suggested activities, allowing family members to quickly execute their activity plans.

[0581] Furthermore, image and video data captured by the device are automatically collected using image processing libraries such as OpenCV and organized chronologically. The organized data is uploaded to a server and used as a digital album displayed chronologically. This album is extremely convenient for users to look back on family memories in real time.

[0582] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0583] Step 1:

[0584] The server receives schedule information from each household member's smartphone. The input data consists of each household member's schedule, which is integrated on the server using the Google Calendar API. The server then stores the received schedule data in a database.

[0585] Step 2:

[0586] The server analyzes the integrated schedule information and checks for duplicates. The input is the integrated schedule data received in step 1, and the server compares and analyzes the time of each scheduled event. If duplicates are found, the server generates a suggested optimal schedule as instructed by the program and outputs the reconstructed schedule.

[0587] Step 3:

[0588] The server uses a generation AI model to suggest activities based on past outing data of family members. The input is past outing logs. The server utilizes AI to analyze and process family interests and past event data. Based on this analysis, it suggests activities suitable for holidays and weekends and outputs them to the terminal.

[0589] Step 4:

[0590] The server executes the online booking process based on the proposed activity. The input is the details of the proposed activity, and the server processes it via access to the booking site and outputs booking confirmation information. The user receives a confirmation message on their terminal.

[0591] Step 5:

[0592] The system automatically uploads images and video data captured by the user's device to the server. The input is digital image data captured by the device. The server uses image processing libraries such as OpenCV to organize the data and outputs a digital album based on chronological order. This digital album can be viewed and checked by the user at any time.

[0593] Step 6:

[0594] The smart glasses, which act as a visual display device, present integrated information to the user. The input consists of integrated schedule and activity information transmitted from a server. The smart glasses visually display and output this information. The user then reviews the schedule based on the displayed information and makes adjustments as needed.

[0595] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0596] This invention is a system that centrally manages the schedules of each member of a household and has a function to propose action plans that combine an emotional engine. This invention makes it possible to provide suggestions and notifications that are tailored to the preferences and emotional states of each individual member.

[0597] Integration of schedule management and emotional engine

[0598] The server receives schedule information entered by the user from the terminal and stores it centrally in a database. When the user inputs or automatically acquires emotional data in their daily life, the terminal utilizes an emotion engine to analyze the user's emotional state. This emotional data is also stored on the server along with other schedule information. The server then proposes an optimal schedule that takes the emotional state into account and provides the content to the user's terminal.

[0599] Specific example:

[0600] If emotional data indicating that the user is tired is recorded, the server will prioritize suggesting activities that promote relaxation. For example, instead of a scheduled strenuous exercise event, it will schedule a quieter activity such as watching a movie or taking a walk, and notify the device.

[0601] Action plan proposal and emotional feedback

[0602] The emotion engine analyzes the user's emotional state in real time based on configured parameters. The server receives this analysis result and uses it to improve the accuracy of activity recommendations. For example, if the emotion is determined to be positive, a challenging activity can be recommended.

[0603] Specific example:

[0604] On days when users are feeling energetic, the server sends notifications to their devices encouraging them to participate in outdoor activities or sporting events, and automatically arranges event reservations if necessary.

[0605] Notification customization that reflects emotional state

[0606] The server provides a feature that customizes the content and method of notifications, taking into account the user's emotional state. This ensures that important schedules and events are notified in a way that is appropriate to the user's emotions, improving the user experience.

[0607] Specific example:

[0608] If the user is feeling stressed, the server generates a message in a softer tone and sends it to the device. Conversely, if the user is relaxed, a notification is sent that includes more specific and action-oriented instructions.

[0609] Adding emotion to digital albums

[0610] Photos and videos taken by users in their daily lives are uploaded to a server along with emotional data and organized not only chronologically but also based on emotional tags. Based on this information, the server creates digital albums that record memories, allowing users to look back on the changes in their emotions.

[0611] As described above, the present invention is a system for realizing schedule management and action suggestions that take emotional elements into consideration, and can provide users with a more personalized experience.

[0612] The following describes the processing flow.

[0613] Step 1:

[0614] The user inputs their daily schedule information through their device, and simultaneously, the emotion engine records their current emotional state data. This allows for the collection of both scheduled and emotional information.

[0615] Step 2:

[0616] The terminal collects schedule information and sentiment data, which are then sent to the server using a secure protocol. The server stores this data in a database and prepares it for the next processing step.

[0617] Step 3:

[0618] The server centrally organizes the schedule information it receives and integrates it as the schedule for all household members. During this process, it takes emotional data into consideration and adjusts the priority and content of the schedule accordingly.

[0619] Step 4:

[0620] The server generates an optimal action plan based on information from the emotion engine. This plan also reflects individual preferences and activities from an existing database.

[0621] Step 5:

[0622] The server generates an action plan, which is then delivered to each user's terminal in real time. The terminal visualizes this plan and presents it in a format that is easy for the user to understand.

[0623] Step 6:

[0624] The user reviews the proposed action plan on their device and provides feedback if necessary. It's also possible to notify the emotion engine of adjustments based on changes in their emotions.

[0625] Step 7:

[0626] The emotion engine performs more accurate analysis based on updated emotion data and user feedback. The server receives the results and reviews the content and method of notifications again.

[0627] Step 8:

[0628] The server prepares emotionally responsive, customized notifications and sends them to the user's device at the appropriate time. For example, stress reduction suggestions might be sent as gentle messages.

[0629] Step 9:

[0630] Photos and videos that users record in their daily lives are linked to emotional information and uploaded to a server via their devices. The server analyzes this data, tagging it with emotions and organizing it accordingly.

[0631] Step 10:

[0632] Based on the data organized by the server, a digital album is created. This album displays the user's emotional changes and is accessible from their device.

[0633] (Example 2)

[0634] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0635] Managing the schedules of multiple family members in modern households is complex, and proposing activity plans that take into account each member's feelings and preferences is difficult. Furthermore, there is a lack of effective ways to organize daily digital content and optimize notifications based on emotional information. This creates a need for a suitable personal assistant system to improve the quality of daily life.

[0636] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0637] In this invention, the server includes means for centrally managing schedule data, means for analyzing emotional states and proposing activity plans based on them, and means for collecting and organizing captured digital content along with emotional information. This enables users to receive personalized suggestions based on their individual emotions and schedules.

[0638] "Schedule data" refers to all information about activities and events that a user plans in their daily life, including time, place, and content.

[0639] "Means of centralized management" refers to methods and processes for centrally aggregating and efficiently handling information obtained from multiple data sources.

[0640] "Emotional state" refers to information that describes the psychological situation or mood a user is feeling at a particular point in time.

[0641] "Means for analyzing the state" refer to algorithms and software functions used to process collected data and understand its meaning and trends.

[0642] "Means of proposing an activity plan" refers to methods and processes for showing users the optimal actions or options based on acquired data.

[0643] "Digital content" refers to information stored in electronic format, including various media formats such as images, videos, and music.

[0644] "Emotional information" refers to data that indicates a user's psychological state, usually expressed as numbers or text, and forms the basis for analyzing user needs.

[0645] "Methods for optimizing notifications" refer to the most effective methods and processes for determining the timing and content of information sent to users, tailored to individual circumstances.

[0646] This invention is a system that manages the schedules and emotions of each member of the household to provide appropriate activity suggestions and notifications to individual users. The system includes a server, terminals, an emotion engine, and a generative AI model as its main components.

[0647] The server is positioned to centrally manage schedule data and emotional information provided by each user from their terminal. Schedule data details the user's daily schedule, while emotional information typically represents their psychological state, which is analyzed by an emotional engine. This information is stored in a database on the server and used to generate customized activity plans for each user.

[0648] Users use devices such as smartphones and tablets to input their daily schedules and manually record their emotions. These devices are equipped with an emotion engine to analyze the user's emotional state, enabling automated emotion analysis. The latest emotion recognition technology is used to collect emotion data, generating highly accurate emotion labels.

[0649] The data collected from the device is further analyzed and activity suggestions are made using a server-based AI model. The AI ​​model combines schedule data and emotional information to generate an optimal plan tailored to the user. The results are provided as notifications to the user's device, enabling flexible activity suggestions that align with their lifestyle.

[0650] For example, on a day when a user feels tired, the server uses a generated AI model to suggest relaxing activities such as "watching a movie" or "listening to music." If the user prompts on their device with "What activities should I do today?", appropriate suggestions will be presented in response to the question, "What activities can be suggested if the user's mood is 'relaxed'?"

[0651] This system configuration allows users to enjoy a personalized, high-quality experience tailored to their emotions and schedule at any given time.

[0652] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0653] Step 1:

[0654] The user enters schedule information and sentiment data into the device.

[0655] Input: Information about daily schedules and current emotional state

[0656] Processing: The terminal receives data entered by the user, converts schedules into calendar data format, and analyzes emotional data through an emotion engine. This generates emotional labels (e.g., fatigue, happiness).

[0657] Output: Organized schedule data and analyzed sentiment labels

[0658] Step 2:

[0659] The device sends data to the server.

[0660] Input: Organized schedule data and sentiment labels

[0661] Processing: The terminal securely transmits the collected data to the server via the network for centralized management.

[0662] Output: User-specific schedule data and sentiment information stored on the server.

[0663] Step 3:

[0664] The server performs analysis based on stored data to generate activity suggestions.

[0665] Input: Schedule data and sentiment information stored on the server

[0666] Processing: The server uses a generative AI model to analyze data and generate activity suggestions optimized for the user. The generative AI model leverages information learned from the user's past activity history and emotional patterns.

[0667] Output: Generated activity suggestions (e.g., suggestion to watch a movie while relaxing)

[0668] Step 4:

[0669] The server notifies the user's terminal of the proposed results.

[0670] Input: Generated activity proposals

[0671] Processing: The server customizes the suggestions based on the user's emotional state, creates a notification, and sends it to the device.

[0672] Output: Customized activity suggestions displayed on the user's terminal.

[0673] Step 5:

[0674] The user selects an action in response to a suggestion and enters feedback about the activity performed into the device.

[0675] Input: Feedback / reactions regarding activities carried out based on the suggestion

[0676] Processing: The terminal receives user feedback and prepares to send it to the server. This data will be used to improve future suggestions.

[0677] Output: User feedback collected for future proposals

[0678] Step 6:

[0679] The server adjusts the generated AI model based on the feedback, improving the accuracy of the next suggestion.

[0680] Input: User feedback data

[0681] Processing: The server analyzes the feedback and updates the parameters of the generated AI model to improve the accuracy of the next activity suggestion.

[0682] Output: Improved suggestion capabilities through a refined AI model.

[0683] (Application Example 2)

[0684] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0685] It is necessary to optimally manage the diverse schedules of each family member and propose action plans that take into account each member's emotional state. Conventional systems lack sufficient activity suggestions and notification customization that reflect emotional states, making effective schedule management difficult. Therefore, there is a need to achieve flexible and personalized schedule adjustments based on emotions.

[0686] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0687] In this invention, the server includes means for centrally managing schedule information, means for proposing action plans based on the emotional state of each household member, means for organizing image and video data based on emotional data, and means for customizing notification methods according to the emotional state. As a result, users can receive action plan suggestions optimized for their emotional state.

[0688] "Schedule information" refers to data about the activities and plans of each household member, and should be managed centrally.

[0689] "Emotional state" refers to data that indicates the psychological and emotional state of family members, and is an important element when proposing action plans.

[0690] An "action plan" refers to daily activities and schedules proposed that take into account the schedules and emotional states of each family member.

[0691] "Emotional data" refers to measurements and indicators that show the emotional state of household members, and is used for organizing image and video data and customizing notifications.

[0692] "Notification method" refers to the means and format for delivering information and suggestions to household members, and should be adjusted according to the user's emotional state.

[0693] The system for implementing this invention consists of a platform including a consumer robot that monitors the schedule and emotional state of each member of the household. The server aggregates schedule information input from the user and then utilizes an emotion engine to perform emotion analysis. The emotion data is analyzed, and an optimal action plan is proposed based on the results. The proposed action plan is notified to the user through the user's terminal or the robot's display or voice output function.

[0694] The system uses the following hardware: a microphone for voice recognition, a camera to enable user emotion analysis, and a display for showing notifications. The software includes a schedule management application, the Emotion API as an emotion analysis AI, and a notification customization algorithm that generates statements. The server aggregates this data and proposes action plans.

[0695] For example, if the server determines that the user is emotionally fatigued, it will suggest activities to help them relax, such as watching a movie or doing some light stretching. This suggestion is communicated in a gentle tone to reduce the user's burden.

[0696] An example of a prompt message is, "Based on the emotional data of all family members, please suggest activities that the family can enjoy together this weekend. Please be specific, including suggestions for relaxing movies and snacks." In this way, users can receive suggestions optimized for their emotional state, enabling them to live a more fulfilling life.

[0697] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0698] Step 1:

[0699] The terminal receives schedule information from the user and sends it to the server. The input is the user's specific schedule, and the output is aggregated schedule data. This data is stored on the server.

[0700] Step 2:

[0701] The server uses an emotion engine to analyze emotion data sent from the user's device. The input is data indicating the user's emotional state, and the output is the result of the analysis of the emotional state. This reveals which emotion is dominant.

[0702] Step 3:

[0703] The server generates an action plan based on aggregated schedule data and analyzed emotional states. The input is the schedule and the results of the emotional analysis, and the output is the optimal action plan proposed to the user. This plan is adjusted according to the emotional state.

[0704] Step 4:

[0705] The terminal receives an action plan from the server and notifies the user. The input is the action plan sent from the server, and the output is the specific notification content for the user. The notification content is delivered to the user via display or audio.

[0706] Step 5:

[0707] The user reviews the proposed action plan and enters feedback into the terminal. The input is the user's feedback, and the output is data used to improve the accuracy of future proposals. This feedback is sent to the server and used for future proposals.

[0708] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0709] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0710] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0711] [Fourth Embodiment]

[0712] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0713] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0714] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0715] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0716] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0717] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0718] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0719] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0720] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0721] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0722] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0723] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0724] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0725] This invention is a system for efficiently managing the lives of the entire family, and has three main functions: schedule management, activity suggestions, and storage of growth records.

[0726] Schedule management

[0727] The server receives schedule information entered by each household member via their device and manages it centrally. This integrates parents' work schedules, children's school events, and extracurricular activity schedules into a single platform. The server checks for duplicates, generates an optimized schedule, and delivers it to the device. Users can then use this to manage their time efficiently.

[0728] Specific example:

[0729] When users enter their school events or work meetings into their devices, the server centralizes them and delivers a coordinated schedule that avoids conflicts. For example, if a child's sports day clashes with an important parental meeting, the server will suggest rescheduling the meeting.

[0730] Activity suggestions

[0731] The server suggests activities suitable for weekends and holidays based on the family members' past data and entered preferences. These suggestions include events selected to be enjoyable for the whole family, and the server can also assist with reservations and ticket arrangements.

[0732] Specific example:

[0733] On a fine weekend, the server suggests local festivals tailored to the family's interests. If necessary, the server automatically purchases admission tickets online and sends a confirmation message to the device.

[0734] Saving growth records

[0735] The device automatically uploads and organizes photos and videos taken by the user in their daily life to a server. This allows the server to create digital albums in chronological order, making it easy for the user to relive precious family memories.

[0736] Specific example:

[0737] Photos and videos taken by users at their children's birthday parties are categorized by event and automatically saved to the cloud. Later, family members can view these memories as digital albums on their devices.

[0738] Thus, the present invention provides a comprehensive system for facilitating the daily lives of families and supports each family member in efficiently managing their schedule and keeping records.

[0739] The following describes the processing flow.

[0740] Step 1:

[0741] Users input their schedule information through a smartphone or PC application. Here, they can add detailed information about work appointments, school events, extracurricular activities, etc.

[0742] Step 2:

[0743] The terminal encrypts the entered schedule information and sends it to the server in the cloud. This ensures that the data is transferred securely.

[0744] Step 3:

[0745] The server stores the received schedule information in a database, centralizing the schedules of each household member. The integrated data is then formatted and structured.

[0746] Step 4:

[0747] The server uses AI algorithms to detect schedule overlaps and conflicts and generates optimized schedule proposals to resolve them.

[0748] Step 5:

[0749] The server generates a pre-arranged schedule and sends it to each household member's device. The device then displays this schedule in a visually easy-to-understand format.

[0750] Step 6:

[0751] Users can review the schedule proposed by the server via their device and make modifications or approvals as needed.

[0752] Step 7:

[0753] The server suggests weekend and holiday activities based on the user's past activity history and interests. This includes booking events and purchasing tickets.

[0754] Step 8:

[0755] Once the user reviews the suggested activities and selects one they are interested in, the server automatically completes the necessary reservations and arranges the tickets.

[0756] Step 9:

[0757] The user saves photos and videos taken in their daily life to their device. These media files are automatically uploaded to the cloud.

[0758] Step 10:

[0759] The server analyzes the uploaded media data, classifying and organizing it based on date, time, and events. It then creates a digital album from this data.

[0760] Step 11:

[0761] The device provides a digital album that is accessible to all family members. Through this, users can revisit family memories anytime.

[0762] (Example 1)

[0763] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0764] In daily family life, individual schedules often overlap and clash, making it difficult to make the most of free time. Furthermore, planning activities that all family members can enjoy requires considerable effort, and organizing memorable photos and videos is a cumbersome task.

[0765] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0766] In this invention, the server includes means for aggregating and managing schedule information entered by household members, means for proposing a life plan using a generation model based on the past activity history and preferences of household members, and means for automatically receiving and classifying image and video information. This makes it possible to optimize the schedules of each household member and automate the proposal and planning of activities that everyone can enjoy.

[0767] The term "family member" refers to the individual people who belong to the same household.

[0768] "Schedule information" refers to data that shows the planned activities of individuals or groups at a specific date and time.

[0769] A "generative model" refers to a technology that uses algorithms or software to automatically generate new suggestions based on past data and input information.

[0770] "Image and video information" refers to a data set that includes digital files in the form of still images and videos that have been captured.

[0771] An "algorithm" is a logical framework for performing a series of steps or computational processes designed to solve a specific problem.

[0772] "Optimization" refers to the operation or process of adjusting something to achieve the best possible result based on certain criteria.

[0773] "Planning" is the process of organizing multiple schedules and activities and creating a plan for their effective implementation.

[0774] This system is designed to efficiently manage the lives of the entire family. Its main functions include schedule management, activity suggestions, and the storage of growth records, all of which are achieved through the cooperation of a server and terminals.

[0775] The server receives schedule information from household members sent from terminals and operates a database to aggregate and manage it. This database incorporates algorithms to detect schedule overlaps and optimize schedules, enabling centralized schedule management.

[0776] Specifically, users input their schedules, such as school events or work meetings, using their devices. The server integrates this information and optimizes the plan using algorithms as needed. For example, if a child's sports day clashes with an important business meeting, the server automatically suggests rescheduling the meeting to a different date and time.

[0777] Furthermore, the server employs a generative AI model that suggests activities based on the past activity history and preferences of household members. This makes it possible to provide optimal weekend plans tailored to conditions such as weather and duration. For example, it can recommend participation in local events suitable for sunny days and provide support for purchasing tickets online if necessary.

[0778] Furthermore, the device has a function that automatically sends information about images and videos taken by the user to the server. The server organizes the received data by event and stores it in the cloud, allowing the user to easily view it on the device at a later date.

[0779] In addition to the above process, the AI ​​model generates a prompt that reads, "Design a platform where the schedules of the whole family are centralized. Include a feature to prevent conflicts between parents' work and children's school events." This enables centralized data management and the suggestion of appropriate activities.

[0780] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0781] Step 1:

[0782] The user uses a terminal to enter schedule information for each household member. This information includes data on school events and work meetings. The terminal converts this schedule information into a specified format and sends it to the server.

[0783] Step 2:

[0784] The server stores the schedule information received from the terminals in a database and processes the data for centralized management of household members' schedules. Here, an algorithm is applied to detect duplicate schedules, enabling efficient management.

[0785] Step 3:

[0786] The server detects duplicate schedule information and uses a generation AI model to reference each household member's past data and preferences to generate an optimal schedule. This process performs data calculations that automate schedule adjustments and suggestions. The generated schedule is delivered to the user's device.

[0787] Step 4:

[0788] Users can view the schedule proposed by the server from their terminal and make manual modifications as needed. The user's modifications are then sent back to the server and updated in the database.

[0789] Step 5:

[0790] The server uses a generative AI model to create activities based on past activity history and preferences, in order to suggest activities suitable for each household member. Data processing is performed here, incorporating external event information, and the suggestions are then notified to the user's device.

[0791] Step 6:

[0792] The device automatically uploads images and videos taken by the user to the server. The server categorizes this media data by event and saves it to cloud storage. This process involves data calculations for file organization and backup.

[0793] Step 7:

[0794] Users can view digital albums stored in the cloud through their devices and relive family memories. This makes it easy to access digital media.

[0795] (Application Example 1)

[0796] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0797] In modern households, managing each member's schedule, suggesting activities, and organizing growth records is extremely complex. This makes it difficult for the whole family to efficiently manage their lives together. In particular, there is a need for activity planning based on the individual preferences and needs of each family member, as well as the effective presentation of visual information. A system is needed to address these challenges and make daily family life smoother and more enjoyable.

[0798] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0799] In this invention, the server includes means for centrally managing schedule information, means for proposing action plans based on the preferences of each household member, means for automatically collecting and organizing image and video data, means for presenting integrated information using a visual display device, means for proposing activities based on past data of outings, means for booking and confirming outings, and means for displaying videos and images in chronological order. This makes it possible for each member of the household to check and flexibly manage each other's schedules and activities in real time.

[0800] "A means of centrally managing schedule information" refers to a system that integrates and manages all of a family member's schedules on a single platform.

[0801] "A means of proposing an action plan based on the preferences of each household member" refers to a function that analyzes the preferences and past activity data of each household member and proposes an action plan that is appropriate for them.

[0802] "Means for automatically collecting and organizing image and video data" refers to the process of automatically uploading photos and videos taken by household members to the cloud, and then organizing and saving them.

[0803] "Means of notifying about specific events or occasions" refers to a system for notifying household members of important events or deadlines on the calendar as they approach.

[0804] "Means of presenting integrated information using a visual display device" refers to a function that visually displays integrated schedules and notifications using devices such as smart glasses.

[0805] "A means of suggesting activities based on past data of outings" refers to a system that suggests optimal future activities based on the past outing history of household members.

[0806] "Means for booking and confirming outings" refers to a process that automatically books suggested activities and allows household members to confirm the necessary information.

[0807] "Means for displaying video and images in chronological order" refers to a function that organizes captured video and images in chronological order and makes them viewable on a visual device.

[0808] The system of this invention aims to efficiently manage the lives of the entire family and support the schedules and action plans of each household member. The system is centered around a server and functions by exchanging information with terminals such as smartphones and smart glasses.

[0809] The server receives and centrally manages schedule information. Schedule information is aggregated using the Google Calendar API, integrating the schedules of household members. This allows for automatic checking of overlapping appointments and the delivery of optimized schedules to devices in real time.

[0810] On user terminals, integrated information is presented using visual display devices (e.g., smart glasses), and important events and schedule changes are notified. This allows household members to easily check their schedules and make decisions smoothly.

[0811] Furthermore, the server uses a generative AI model to suggest activities based on the family members' past outing data. For example, by analyzing preferences for places and events visited in the past, it can suggest new activities based on prompts such as, "Based on the family's past holiday activity data, please suggest the best activities for next weekend." The server also handles booking and confirmation of suggested activities, allowing family members to quickly execute their activity plans.

[0812] Furthermore, image and video data captured by the device are automatically collected using image processing libraries such as OpenCV and organized chronologically. The organized data is uploaded to a server and used as a digital album displayed chronologically. This album is extremely convenient for users to look back on family memories in real time.

[0813] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0814] Step 1:

[0815] The server receives schedule information from the smartphones of household members. The input data consists of each household member's schedule, which is integrated on the server using the Google Calendar API. The server then stores the received schedule data in a database.

[0816] Step 2:

[0817] The server analyzes the integrated schedule information and checks for duplicates. The input is the integrated schedule data received in step 1, and the server compares and analyzes the time of each scheduled event. If duplicates are found, the server generates a suggested optimal schedule as instructed by the program and outputs the reconstructed schedule.

[0818] Step 3:

[0819] The server uses a generation AI model to suggest activities based on past outing data of family members. The input is past outing logs. The server utilizes AI to analyze and process family interests and past event data. Based on this analysis, it suggests activities suitable for holidays and weekends and outputs them to the terminal.

[0820] Step 4:

[0821] The server executes the online booking process based on the proposed activity. The input is the details of the proposed activity, and the server processes it via access to the booking site and outputs booking confirmation information. The user receives a confirmation message on their terminal.

[0822] Step 5:

[0823] The system automatically uploads images and video data captured by the user's device to the server. The input is digital image data captured by the device. The server uses image processing libraries such as OpenCV to organize the data and outputs a digital album based on chronological order. This digital album can be viewed and checked by the user at any time.

[0824] Step 6:

[0825] The smart glasses, which act as a visual display device, present integrated information to the user. The input consists of integrated schedule and activity information transmitted from a server. The smart glasses visually display and output this information. The user then reviews the schedule based on the displayed information and makes adjustments as needed.

[0826] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0827] This invention is a system that centrally manages the schedules of each member of a household and has a function to propose action plans that combine an emotional engine. This invention makes it possible to provide suggestions and notifications that are tailored to the preferences and emotional states of each individual member.

[0828] Integration of schedule management and emotional engine

[0829] The server receives schedule information entered by the user from the terminal and stores it centrally in a database. When the user inputs or automatically acquires emotional data in their daily life, the terminal utilizes an emotion engine to analyze the user's emotional state. This emotional data is also stored on the server along with other schedule information. The server then proposes an optimal schedule that takes the emotional state into account and provides the content to the user's terminal.

[0830] Specific example:

[0831] If emotional data indicating that the user is tired is recorded, the server will prioritize suggesting activities that promote relaxation. For example, instead of a scheduled strenuous exercise event, it will schedule a quieter activity such as watching a movie or taking a walk, and notify the device.

[0832] Action plan proposal and emotional feedback

[0833] The emotion engine analyzes the user's emotional state in real time based on configured parameters. The server receives this analysis result and uses it to improve the accuracy of activity recommendations. For example, if the emotion is determined to be positive, a challenging activity can be recommended.

[0834] Specific example:

[0835] On days when users are feeling energetic, the server sends notifications to their devices encouraging them to participate in outdoor activities or sporting events, and automatically arranges event reservations if necessary.

[0836] Notification customization that reflects emotional state

[0837] The server provides a feature that customizes the content and method of notifications, taking into account the user's emotional state. This ensures that important schedules and events are notified in a way that is appropriate to the user's emotions, improving the user experience.

[0838] Specific example:

[0839] If the user is feeling stressed, the server generates a message in a softer tone and sends it to the device. Conversely, if the user is relaxed, a notification is sent that includes more specific and action-oriented instructions.

[0840] Adding emotion to digital albums

[0841] Photos and videos taken by users in their daily lives are uploaded to a server along with emotional data and organized not only chronologically but also based on emotional tags. Based on this information, the server creates digital albums that record memories, allowing users to look back on the changes in their emotions.

[0842] As described above, the present invention is a system for realizing schedule management and action suggestions that take emotional elements into consideration, and can provide users with a more personalized experience.

[0843] The following describes the processing flow.

[0844] Step 1:

[0845] The user inputs their daily schedule information through their device, and simultaneously, the emotion engine records their current emotional state data. This allows for the collection of both scheduled and emotional information.

[0846] Step 2:

[0847] The terminal collects schedule information and sentiment data, which are then sent to the server using a secure protocol. The server stores this data in a database and prepares it for the next processing step.

[0848] Step 3:

[0849] The server centrally organizes the schedule information it receives and integrates it as the schedule for all household members. During this process, it takes emotional data into consideration and adjusts the priority and content of the schedule accordingly.

[0850] Step 4:

[0851] The server generates an optimal action plan based on information from the emotion engine. This plan also reflects individual preferences and activities from an existing database.

[0852] Step 5:

[0853] The server generates an action plan, which is then delivered to each user's terminal in real time. The terminal visualizes this plan and presents it in a format that is easy for the user to understand.

[0854] Step 6:

[0855] The user reviews the proposed action plan on their device and provides feedback if necessary. It's also possible to notify the emotion engine of adjustments based on changes in their emotions.

[0856] Step 7:

[0857] The emotion engine performs more accurate analysis based on updated emotion data and user feedback. The server receives the results and reviews the content and method of notifications again.

[0858] Step 8:

[0859] The server prepares emotionally responsive, customized notifications and sends them to the user's device at the appropriate time. For example, stress reduction suggestions might be sent as gentle messages.

[0860] Step 9:

[0861] Photos and videos that users record in their daily lives are linked to emotional information and uploaded to a server via their devices. The server analyzes this data, tagging it with emotions and organizing it accordingly.

[0862] Step 10:

[0863] Based on the data organized by the server, a digital album is created. This album displays the user's emotional changes and is accessible from their device.

[0864] (Example 2)

[0865] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0866] Managing the schedules of multiple family members in modern households is complex, and proposing activity plans that take into account each member's feelings and preferences is difficult. Furthermore, there is a lack of effective ways to organize daily digital content and optimize notifications based on emotional information. This creates a need for a suitable personal assistant system to improve the quality of daily life.

[0867] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0868] In this invention, the server includes means for centrally managing schedule data, means for analyzing emotional states and proposing activity plans based on them, and means for collecting and organizing captured digital content along with emotional information. This enables users to receive personalized suggestions based on their individual emotions and schedules.

[0869] "Schedule data" refers to all information about activities and events that a user plans in their daily life, including time, place, and content.

[0870] "Means of centralized management" refers to methods and processes for centrally aggregating and efficiently handling information obtained from multiple data sources.

[0871] "Emotional state" refers to information that describes the psychological situation or mood a user is feeling at a particular point in time.

[0872] "Means for analyzing the state" refer to algorithms and software functions used to process collected data and understand its meaning and trends.

[0873] "Means of proposing an activity plan" refers to methods and processes for showing users the optimal actions or options based on acquired data.

[0874] "Digital content" refers to information stored in electronic format, including various media formats such as images, videos, and music.

[0875] "Emotional information" refers to data that indicates a user's psychological state, usually expressed as numbers or text, and forms the basis for analyzing user needs.

[0876] "Methods for optimizing notifications" refer to the most effective methods and processes for determining the timing and content of information sent to users, tailored to individual circumstances.

[0877] This invention is a system that manages the schedules and emotions of each member of the household to provide appropriate activity suggestions and notifications to individual users. The system includes a server, terminals, an emotion engine, and a generative AI model as its main components.

[0878] The server is positioned to centrally manage schedule data and emotional information provided by each user from their terminal. Schedule data details the user's daily schedule, while emotional information typically represents their psychological state, which is analyzed by an emotional engine. This information is stored in a database on the server and used to generate customized activity plans for each user.

[0879] Users use devices such as smartphones and tablets to input their daily schedules and manually record their emotions. These devices are equipped with an emotion engine to analyze the user's emotional state, enabling automated emotion analysis. The latest emotion recognition technology is used to collect emotion data, generating highly accurate emotion labels.

[0880] The data collected from the device is further analyzed and activity suggestions are made using a server-based AI model. The AI ​​model combines schedule data and emotional information to generate an optimal plan tailored to the user. The results are provided as notifications to the user's device, enabling flexible activity suggestions that align with their lifestyle.

[0881] For example, on a day when a user feels tired, the server uses a generated AI model to suggest relaxing activities such as "watching a movie" or "listening to music." If the user prompts on their device with "What activities should I do today?", appropriate suggestions will be presented in response to the question, "What activities can be suggested if the user's mood is 'relaxed'?"

[0882] This system configuration allows users to enjoy a personalized, high-quality experience tailored to their emotions and schedule at any given time.

[0883] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0884] Step 1:

[0885] The user enters schedule information and sentiment data into the device.

[0886] Input: Information about daily schedules and current emotional state

[0887] Processing: The terminal receives data entered by the user, converts schedules into calendar data format, and analyzes emotional data through an emotion engine. This generates emotional labels (e.g., fatigue, happiness).

[0888] Output: Organized schedule data and analyzed sentiment labels

[0889] Step 2:

[0890] The device sends data to the server.

[0891] Input: Organized schedule data and sentiment labels

[0892] Processing: The terminal securely transmits the collected data to the server via the network for centralized management.

[0893] Output: User-specific schedule data and sentiment information stored on the server.

[0894] Step 3:

[0895] The server performs analysis based on stored data to generate activity suggestions.

[0896] Input: Schedule data and sentiment information stored on the server

[0897] Processing: The server uses a generative AI model to analyze data and generate activity suggestions optimized for the user. The generative AI model leverages information learned from the user's past activity history and emotional patterns.

[0898] Output: Generated activity suggestions (e.g., suggestion to watch a movie while relaxing)

[0899] Step 4:

[0900] The server notifies the user's terminal of the proposed results.

[0901] Input: Generated activity proposals

[0902] Processing: The server customizes the suggestions based on the user's emotional state, creates a notification, and sends it to the device.

[0903] Output: Customized activity suggestions displayed on the user's terminal.

[0904] Step 5:

[0905] The user selects an action in response to a suggestion and enters feedback about the activity performed into the device.

[0906] Input: Feedback / reactions regarding activities carried out based on the suggestion

[0907] Processing: The terminal receives user feedback and prepares to send it to the server. This data will be used to improve future suggestions.

[0908] Output: User feedback collected for future proposals

[0909] Step 6:

[0910] The server adjusts the generated AI model based on the feedback, improving the accuracy of the next suggestion.

[0911] Input: User feedback data

[0912] Processing: The server analyzes the feedback and updates the parameters of the generated AI model to improve the accuracy of the next activity suggestion.

[0913] Output: Improved suggestion capabilities through a refined AI model.

[0914] (Application Example 2)

[0915] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0916] It is necessary to optimally manage the diverse schedules of each family member and propose action plans that take into account each member's emotional state. Conventional systems lack sufficient activity suggestions and notification customization that reflect emotional states, making effective schedule management difficult. Therefore, there is a need to achieve flexible and personalized schedule adjustments based on emotions.

[0917] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0918] In this invention, the server includes means for centrally managing schedule information, means for proposing action plans based on the emotional state of each household member, means for organizing image and video data based on emotional data, and means for customizing notification methods according to the emotional state. As a result, users can receive action plan suggestions optimized for their emotional state.

[0919] "Schedule information" refers to data about the activities and plans of each household member, and should be managed centrally.

[0920] "Emotional state" refers to data that indicates the psychological and emotional state of family members, and is an important element when proposing action plans.

[0921] An "action plan" refers to daily activities and schedules proposed that take into account the schedules and emotional states of each family member.

[0922] "Emotional data" refers to measurements and indicators that show the emotional state of household members, and is used for organizing image and video data and customizing notifications.

[0923] "Notification method" refers to the means and format for delivering information and suggestions to household members, and should be adjusted according to the user's emotional state.

[0924] The system for implementing this invention consists of a platform including a consumer robot that monitors the schedule and emotional state of each member of the household. The server aggregates schedule information input from the user and then utilizes an emotion engine to perform emotion analysis. The emotion data is analyzed, and an optimal action plan is proposed based on the results. The proposed action plan is notified to the user through the user's terminal or the robot's display or voice output function.

[0925] The system uses the following hardware: a microphone for voice recognition, a camera to enable user emotion analysis, and a display for showing notifications. The software includes a schedule management application, the Emotion API as an emotion analysis AI, and a notification customization algorithm that generates statements. The server aggregates this data and proposes action plans.

[0926] For example, if the server determines that the user is emotionally fatigued, it will suggest activities to help them relax, such as watching a movie or doing some light stretching. This suggestion is communicated in a gentle tone to reduce the user's burden.

[0927] An example of a prompt message is, "Based on the emotional data of all family members, please suggest activities that the family can enjoy together this weekend. Please be specific, including suggestions for relaxing movies and snacks." In this way, users can receive suggestions optimized for their emotional state, enabling them to live a more fulfilling life.

[0928] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0929] Step 1:

[0930] The terminal receives schedule information from the user and sends it to the server. The input is the user's specific schedule, and the output is aggregated schedule data. This data is stored on the server.

[0931] Step 2:

[0932] The server uses an emotion engine to analyze emotion data sent from the user's device. The input is data indicating the user's emotional state, and the output is the result of the analysis of the emotional state. This reveals which emotion is dominant.

[0933] Step 3:

[0934] The server generates an action plan based on aggregated schedule data and analyzed emotional states. The input is the schedule and the results of the emotional analysis, and the output is the optimal action plan proposed to the user. This plan is adjusted according to the emotional state.

[0935] Step 4:

[0936] The terminal receives an action plan from the server and notifies the user. The input is the action plan sent from the server, and the output is the specific notification content for the user. The notification content is delivered to the user via display or audio.

[0937] Step 5:

[0938] The user reviews the proposed action plan and enters feedback into the terminal. The input is the user's feedback, and the output is data used to improve the accuracy of future proposals. This feedback is sent to the server and used for future proposals.

[0939] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0940] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0941] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0942] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0943] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0944] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0945] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0946] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0947] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0948] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0949] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0950] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0951] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0952] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0953] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0954] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0955] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0956] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0957] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0958] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0959] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0960] The following is further disclosed regarding the embodiments described above.

[0961] (Claim 1)

[0962] A means of centrally managing schedule information,

[0963] A means of proposing an action plan based on the preferences of each household member,

[0964] A means for automatically collecting and organizing image and video data,

[0965] A system that includes means for providing notifications about specific events or occasions.

[0966] (Claim 2)

[0967] The system according to claim 1, further comprising means for automating reservations and purchases based on proposed action plans.

[0968] (Claim 3)

[0969] The system according to claim 1, further comprising means for analyzing the consistency of schedule information and creating an optimized plan.

[0970] "Example 1"

[0971] (Claim 1)

[0972] A means of aggregating and managing schedule information entered by household members,

[0973] A method for proposing a life plan using a generative model based on the past activity history and preferences of household members,

[0974] A means for automatically receiving and classifying image and video information,

[0975] Means for transmitting information regarding a designated event or activity,

[0976] A system that includes means for planning schedule information by applying algorithms.

[0977] (Claim 2)

[0978] The system according to claim 1, further comprising means for performing reservation and purchase operations based on an automatically generated lifestyle plan.

[0979] (Claim 3)

[0980] The system according to claim 1, comprising a function for evaluating the consistency of schedule information and creating an optimized plan.

[0981] "Application Example 1"

[0982] (Claim 1)

[0983] A means of centrally managing schedule information,

[0984] A means of proposing an action plan based on the preferences of each household member,

[0985] A means for automatically collecting and organizing image and video data,

[0986] Means of providing notifications for specific events or occasions,

[0987] A means for presenting integrated information using a visual display device,

[0988] A means of suggesting activities based on past data on outdoor activities,

[0989] A means of booking and confirming outings,

[0990] A system including means for displaying video and images in chronological order.

[0991] (Claim 2)

[0992] The system according to claim 1, further comprising means for automating reservations and purchases based on proposed action plans.

[0993] (Claim 3)

[0994] The system according to claim 1, further comprising means for analyzing the consistency of schedule information and creating an optimized plan.

[0995] "Example 2 of combining an emotion engine"

[0996] (Claim 1)

[0997] A means of centrally managing schedule data,

[0998] A means of analyzing emotional states and proposing an action plan based on that analysis,

[0999] A means of collecting and organizing captured digital content along with emotional information,

[1000] Means to customize notifications for specific appointments or activities,

[1001] A means of adjusting notification content based on emotional information,

[1002] A means of visualizing emotional changes based on historical information,

[1003] A system that includes this.

[1004] (Claim 2)

[1005] The system according to claim 1, further comprising means for automating reservations and purchases by linking with external services based on proposed activity plans.

[1006] (Claim 3)

[1007] The system according to claim 1, further comprising means for analyzing the consistency between schedule data and emotional information to create an activity plan optimized for the user.

[1008] "Application example 2 when combining with an emotional engine"

[1009] (Claim 1)

[1010] A means of centrally managing schedule information,

[1011] A means of proposing an action plan based on the emotional state of each household member,

[1012] A method for organizing image and video data based on emotional data,

[1013] A means to customize notification methods according to emotional state,

[1014] A system that includes means for providing notifications about specific events or occasions.

[1015] (Claim 2)

[1016] The system according to claim 1, comprising means for automating reservations and purchases based on proposed action plans.

[1017] (Claim 3)

[1018] The system according to claim 1, comprising means for analyzing emotional data and optimizing the consistency of schedule information. [Explanation of symbols]

[1019] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of centrally managing schedule information, A means of proposing an action plan based on the preferences of each household member, A means for automatically collecting and organizing image and video data, Means of providing notifications for specific events or occasions, A means for presenting integrated information using a visual display device, A means of suggesting activities based on past data on outdoor activities, A means of booking and confirming outings, A system including means for displaying video and images in chronological order.

2. The system according to claim 1, further comprising means for automating reservations and purchases based on proposed action plans.

3. The system according to claim 1, further comprising means for analyzing the consistency of schedule information and creating an optimized plan.

Images