system

Abstract

Provide a system. 【Solution means】 An input device that acquires voice or text data from a user, An analysis device that analyzes the acquired data to identify the user's emotional state and needs, A generation device that generates a customized action plan based on the user's emotional state and needs, An output device that presents the generated action plan to the user, An assistance device that provides communication guidelines to support the improvement of the user's communication skills, A legal assistance device that provides information on legal procedures, A cooperation device that aims to cooperate with the local community related to the user, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society, procedures related to end-of-life affairs are diverse and complex, making it difficult for individual users to carry out these procedures efficiently and positively. In particular, users often feel an emotional burden and often lack knowledge about legal procedures. As a result, there is a problem that they cannot engage in end-of-life affairs with confidence.

Means for Solving the Problems

[0005] This invention provides a system that analyzes a user's voice or text data to identify the user's emotional state and needs, thereby providing an action plan optimized for the user. Specifically, an input device acquires data from the user, an analysis device analyzes that data to identify the user's emotional state and needs, a generation device generates a customized action plan based on the identified information, and an output device presents this to the user. Furthermore, by including a support device that provides communication guidelines, a legal support device that provides legal assistance, and a collaboration device that facilitates cooperation with local communities, the system realizes an environment in which users can confidently engage in end-of-life planning.

[0006] An "input device" is a device that acquires voice or text data from a user.

[0007] An "analysis device" is a device that analyzes acquired user data to identify their emotional state and needs.

[0008] A "generation device" is a device that generates a customized action plan based on the analyzed data.

[0009] An "output device" is a device that presents the generated action plan to the user.

[0010] A "support device" is a device that provides communication guidelines to help users improve their conversational skills.

[0011] A "legal support device" is a device that provides information about legal procedures.

[0012] A "connection device" is a device designed to facilitate connection between users and their local communities. [Brief explanation of the drawing]

[0013] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0014] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0017] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0018] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0019] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0021] [First Embodiment]

[0022] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0023] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0026] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0029] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0033] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0034] The system of the present invention comprises an input device for handling user voice or text data, an analysis device for analyzing user data, a generation device for generating an action plan based on the analysis, an output device for presenting the plan, a support device for assisting dialogue skills, a legal support device for assisting legal procedures, and a collaboration device for coordinating with local communities. These devices work together to provide users with appropriate end-of-life support.

[0035] Program processing and specific examples

[0036] This indicates the user's intention to begin end-of-life planning.

[0037] The device captures the audio when the user states that they want to begin end-of-life planning, and sends it to the server for analysis.

[0038] The server converts this audio data into text and then performs natural language processing to identify the user's emotions and specific needs. For example, if a user says, "I want to write a will, but I don't know where to start," the analysis system detects that the user is feeling anxious.

[0039] Generating an optimized action plan

[0040] Based on the analysis, the server generates an action plan tailored to the user's situation. This plan details the necessary steps and procedures. For example, if the user wishes to create a will, the plan will specifically outline the legal requirements and procedures for doing so.

[0041] Presentation of action plans and communication support

[0042] The device presents the generated action plan to the user. If the user asks, "What exactly do I need to do?", the output device provides detailed steps. If the user needs assistance communicating with a parent, the support device advises on effective communication methods.

[0043] Support for legal procedures and community engagement

[0044] If the server determines that legal assistance is needed, it will provide legal information appropriate to the user's region through the legal assistance device. This may include details on necessary documents and procedures.

[0045] The connected device retrieves information on local end-of-life planning seminars and events based on the user's location and supports users who wish to participate. For example, it notifies users via their device that "there is a seminar being held at a nearby community center next week."

[0046] Thus, this invention provides services tailored to the specific needs and circumstances of users, creating a system that allows them to approach end-of-life planning with peace of mind.

[0047] The following describes the processing flow.

[0048] Step 1:

[0049] The device captures the user's voice expressing their end-of-life plans and collects the audio data. It verifies that the audio was captured correctly and prompts the user to re-enter any missing information.

[0050] Step 2:

[0051] The terminal instantly converts the captured audio data into text data and sends the data to the server.

[0052] Step 3:

[0053] The server analyzes text data using a natural language processing engine to identify the user's emotions and specific end-of-life planning needs. For example, it can determine if the user wishes to create a will or have their assets managed.

[0054] Step 4:

[0055] The server generates an appropriate action plan using a generator based on identified emotions and needs. This plan is customized according to the user's needs.

[0056] Step 5:

[0057] The device presents the generated action plan to the user in a viewable format. The user can review it via audio or text.

[0058] Step 6:

[0059] When a user asks about details of an action plan or a specific task, the device sends the question to the server to retrieve additional information.

[0060] Step 7:

[0061] The server provides relevant legal procedure information via a legal assistance device, based on the user's place of residence and needs. It presents the user with an overview of the necessary documents and procedures.

[0062] Step 8:

[0063] The server searches for information related to collaboration with local communities and retrieves information on end-of-life planning seminars and events using the linked device.

[0064] Step 9:

[0065] The device notifies the user of acquired local event information and confirms their willingness to participate. With the user's consent, it assists with registration for the event.

[0066] (Example 1)

[0067] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0068] In modern society, a comprehensive support system is needed to assist users in their end-of-life planning process. In particular, a system is required that can generate customized plans based on the user's psychological state, provide support for legal procedures, and facilitate collaboration with local communities. However, a consistent system that meets these requirements does not yet exist. This invention aims to solve these problems.

[0069] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0070] In this invention, the server includes receiving means for acquiring information from the user, analyzing means for analyzing the acquired information to identify the user's psychological state and requests, and generating means for generating an individualized support plan based on the user's psychological state and requests. This enables the user to receive personalized support tailored to their own state and needs. Furthermore, by updating the generated plan, dynamic support can be provided that responds to changes in the user's situation.

[0071] "Receiving means" refers to a device or method for acquiring audio or text information transmitted by a user.

[0072] "Analysis means" refers to a technology or process that analyzes acquired data to identify the user's psychological state and needs.

[0073] "Generation method" refers to a technology or process for creating a customized, individualized response plan for the user based on the analysis results.

[0074] "Display means" refers to a device or technology for visualizing and providing generated plans and information to the user.

[0075] "Support measures" refer to processes that have the function of providing various guidance policies in order to improve users' communication skills.

[0076] "Legal support measures" refer to methods or devices for providing information about legal procedures and for providing the legal support that users need.

[0077] "Collaboration tools" refer to technologies or processes that enable users to collaborate with local societies and activities in which they are involved.

[0078] The "update mechanism" is a process for updating the generated action plan in real time in response to additional user requests or changes in circumstances.

[0079] "Natural language processing technology" is an artificial intelligence technology that interprets human language through data analysis and extracts useful information.

[0080] A "generative AI model" is an artificial intelligence model used to achieve personalized responses and automation in user data analysis and plan generation.

[0081] This invention provides a comprehensive system to support users in their end-of-life planning. This system is designed with multiple devices and means working together.

[0082] First, the user inputs their request into the device via voice or text. At this stage, the voice data is converted into digital data. The device then sends the collected data to a server. The server uses natural language processing technology to analyze this data. Here, a generative AI model is used to extract the user's psychological state and specific needs.

[0083] Based on the analysis results, the server generates a personalized plan best suited to the user. This plan details the necessary procedures and steps, serving as a guide for the user's actions.

[0084] The generated action plan is presented to the user via the device. If the user has questions such as, "What exactly should I do?", the display will provide clear and detailed instructions.

[0085] Furthermore, if a user desires assistance with how to interact with others, the support system provides guidance on effective communication. If information on legal procedures is requested, the legal support system provides legal information relevant to the local area.

[0086] The collaborative method involves using users' local information to collect information on community activities and notifying users, thereby fostering cooperation with local communities.

[0087] A concrete example would be a situation where a user has a request such as, "I want to create a will, but I don't know where to start." In this case, an example of a prompt message to the generating AI model would be, "The user wants to know how to create a will. Please provide advice on where to begin."

[0088] This system provides users with flexible and personalized support tailored to their specific situation, playing a role in alleviating anxieties and doubts related to end-of-life planning.

[0089] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0090] Step 1:

[0091] The user inputs their end-of-life planning requests into the terminal via voice or text. These inputs are specific requests, such as "I want to create a will." The terminal converts the received voice data into a digital format and sends the resulting data to the service server.

[0092] Step 2:

[0093] The server receives digital audio or text data sent from the terminal for analysis. Based on this input data, it uses natural language processing techniques to identify the user's psychological state and needs. This analysis involves data processing, such as converting audio data into text data. Next, it prepares to input the identified psychological state and needs as output into the generating AI model.

[0094] Step 3:

[0095] The server uses an AI model generated based on the analysis results to create a personalized response plan tailored to the user's needs. The input for this process is the user's state information obtained from the analysis results. Based on this, data calculations are performed to output a customized action plan. This action plan includes the necessary procedures and steps, outlining the next actions the user should take.

[0096] Step 4:

[0097] The terminal receives the action plan sent from the server and presents it to the user through a display device. In this step, the received action plan becomes input and is output visually or audibly through the user interface.

[0098] Step 5:

[0099] Users can refer to the presented action plan and then request further assistance with details. For example, they might input a question like, "How can I communicate with my parents?" Based on this input, the support system generates and outputs effective communication advice. This process allows users to learn practical communication methods.

[0100] Step 6:

[0101] If deemed necessary, the server will provide localized legal information through legal assistance tools in response to the user's request regarding legal proceedings. This will include specific details on legally required documents and procedures, and output will be provided to assist the user in preparing their legal case.

[0102] Step 7:

[0103] The integration method involves obtaining information about end-of-life planning-related events taking place in the user's area based on their location information and notifying the user via their device. In this step, local information is input, and based on that information, information about related events is output and communicated to the user.

[0104] (Application Example 1)

[0105] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0106] When elderly people and their families are planning for the end of their lives or their future, they often feel anxious due to the complexity of information gathering and procedures. In particular, information on legal procedures and local support services is difficult to understand, and options for care are limited. Furthermore, there is a problem in that information on activities to improve quality of life is not effectively communicated. To solve these problems, a system is needed that flexibly provides users with appropriate and individualized information and support.

[0107] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0108] In this invention, the server includes a means for collecting and supporting participation in useful local activities based on the user's location, a means for providing a personalized life planning plan based on the analysis results and improving convenience, and an input means for acquiring the user's voice or text data. This makes it possible to provide personalized end-of-life planning and life planning support to the elderly and their families.

[0109] "Audio or text data" refers to information obtained from the user, including recorded audio and information expressed in text.

[0110] "Input means" refers to devices or interfaces for acquiring voice data or text data from users.

[0111] "Analysis means" refers to devices and algorithms that process input data to identify the user's emotional state and needs.

[0112] "Generation means" refers to devices and algorithms for creating an action plan optimized for the user based on the analysis results.

[0113] "Output means" refers to devices or interfaces used to present the generated action plan to the user.

[0114] "Support tools" refer to devices that provide guidance policies and support functions to improve users' conversational skills and communication abilities.

[0115] "Legal support measures" refer to devices that provide users with information and advice related to legal procedures they require.

[0116] "Collaboration tools" refer to devices that build relationships between users and their local communities, and provide useful information and opportunities for participation.

[0117] "Care support tools" refer to devices and functions that provide support for users' life planning and caregiving needs.

[0118] This invention is a system for providing care support, which uses voice or text data from the user to provide personalized support for end-of-life planning and life planning. The system consists of a device such as a smartphone or tablet and a server in the cloud.

[0119] The server uses speech recognition technology to convert the user's voice data into text, utilizing Google® Cloud Speech-to-Text. This converted text data is then processed using the Google Cloud Natural Language API to analyze the user's emotional state and needs. Based on the analyzed information, an optimal action plan tailored to the user's situation is generated. This plan is customized using a generative AI model in the cloud.

[0120] The generated action plan is presented to the user via their device, providing specific procedures, legal information, and information on local community activities. This helps users reduce anxiety and effectively prepare for end-of-life planning and long-term care.

[0121] For example, if a user is considering creating a will as part of "future preparations," their voice, such as "I want to write a will," is captured on the device and analyzed on the server. As a result, an action plan is generated that includes the legal requirements for creating a will and information on seminars held in the area, and this plan is presented to the user. Examples of prompts include, "What should I start with as preparations for the future?" or "Please tell me if there are any end-of-life planning events in my area."

[0122] Thus, the present invention provides a system that offers appropriate and personalized support to users, helping elderly people and their families to plan their lives with peace of mind.

[0123] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0124] Step 1:

[0125] The device acquires voice data from the user. The user inputs questions and wishes regarding end-of-life planning via voice through their smartphone or tablet. This acquired voice data is then sent directly to the server.

[0126] Step 2:

[0127] The server converts received audio data into text data using Google Cloud Speech-to-Text. The input is audio data, and the output is text data in which the audio has been converted into written information. This converted text data is then used for further analysis.

[0128] Step 3:

[0129] The server performs natural language processing on the transcribed data using the Google Cloud Natural Language API. In this step, it receives text data as input and extracts the user's emotional state and specific needs. The output is information indicating the user's emotional analysis results and specific needs.

[0130] Step 4:

[0131] The server generates an action plan using a generative AI model based on information obtained through natural language processing. The input here is the analysis results regarding the user's emotions and needs, and the output is a customized action plan for the user. This action plan can include specific legal procedures and information on local events.

[0132] Step 5:

[0133] The terminal displays an action plan received from the server to the user. The user can then review this plan and decide on specific actions to take regarding their end-of-life planning. The output is the action plan displayed on the terminal in a format that the user can directly view.

[0134] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0135] The system of the present invention comprises an input device and an analysis device for acquiring and analyzing user voice or text data. The input device captures the user's voice input, and the analysis device analyzes this data using a natural language processing engine and an emotion engine. The natural language processing engine understands the specific needs and requests spoken by the user, and the emotion engine identifies the user's emotional state from the context.

[0136] Program processing and specific examples

[0137] Recognizing the user's emotional state

[0138] When a device receives audio from a user, the server decodes the audio data and converts it into text data. For example, audio data such as "I've been stressed a lot lately and I'm worried" is converted into text.

[0139] The server uses a natural language processing engine to interpret user needs and simultaneously uses an emotion engine to recognize emotions such as "stress" and "worry."

[0140] Generating and adjusting action plans

[0141] The server generates a customized action plan based on analyzed needs, while also considering the user's emotional state.

[0142] The content of the action plan is adjusted according to the emotions recognized by the emotion engine. For example, a plan incorporating relaxation techniques will be suggested to a user who is feeling stressed.

[0143] Presenting plans to users and providing feedback.

[0144] The device presents the created action plan to the user visually or audibly. The user is also notified that they can ask additional questions or provide feedback on the content.

[0145] When a user provides feedback on a plan, that information is sent back to the server and used to generate future plans.

[0146] In this way, this system accurately understands the user's emotions and provides support plans accordingly, thereby reducing the anxiety and stress that users experience regarding end-of-life planning and providing comprehensive support.

[0147] The following describes the processing flow.

[0148] Step 1:

[0149] The user expresses their wishes and feelings regarding end-of-life planning in voice, and this voice data is captured by the device. The device receives this voice data, performs initial processing, and sends it to the server.

[0150] Step 2:

[0151] The server converts the received audio data into text data. The converted text is then input into the natural language processing engine and the emotion engine.

[0152] Step 3:

[0153] The server uses a natural language processing engine to extract user needs from the text. Simultaneously, an emotion engine analyzes the text to identify the user's emotional state. For example, it assesses levels of "anxiety," "stress," and "interest."

[0154] Step 4:

[0155] The server generates a personalized action plan based on the analyzed data. The action plan is adjusted according to the user's emotional state, as identified by the emotion engine. For example, a plan including relaxation techniques will be created for a user experiencing anxiety.

[0156] Step 5:

[0157] The device presents the generated action plan to the user. The user can choose whether to view it as text or receive it as an audio notification.

[0158] Step 6:

[0159] Users can review the presented action plan and provide feedback as needed. This feedback is sent to the server and used to improve or adjust the action plan.

[0160] Step 7:

[0161] The server stores user feedback in a database and considers that information when generating the next plan, preparing to provide more personalized support.

[0162] (Example 2)

[0163] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0164] In modern society, users often experience a variety of stresses and anxieties in their daily lives, creating a need for systems that provide appropriate support immediately. However, systems that can quickly and accurately recognize emotional states and specific needs and provide individualized support based on that information are still insufficient. Furthermore, more advanced analytical tools are needed to effectively utilize user feedback and provide customized plans for each user.

[0165] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0166] In this invention, the server includes acquisition means for acquiring voice and text data from the user, analysis means for analyzing the acquired data to identify the user's emotional state and needs, and feedback analysis means for receiving user feedback and utilizing that information to generate future plans. This makes it possible to quickly provide the user with an optimal action plan that is tailored to their emotions and needs, thereby improving the user's quality of life (QOL).

[0167] "Acquisition means" refers to a device or process for receiving voice or text data from a user and incorporating it into the system.

[0168] "Analysis means" refers to a device or method for analyzing acquired data to identify the user's emotional state and specific needs.

[0169] "Generation means" refers to an apparatus or process for creating a customized action plan tailored to the user based on the analysis results.

[0170] "Presentation means" refers to a device or method for presenting the generated action plan to the user visually or audibly.

[0171] A "feedback analysis means" is a device or system that receives feedback from users and utilizes that information to generate future plans.

[0172] "Conversion means" refers to a device or process that converts audio data into text data and prepares it for further analysis.

[0173] An "update mechanism" refers to a device or process for updating the action plan presented to the user in real time and making adjustments based on the user's situation and feedback.

[0174] This invention relates to a system that acquires and analyzes user voice or text data to provide a customized action plan tailored to the user's emotional state and specific needs. The system comprises acquisition means, analysis means, generation means, presentation means, feedback analysis means, and conversion means.

[0175] The server has a means of receiving audio from the user, and this audio data is converted into a digital format. The hardware used can be a device with a built-in microphone or an external microphone. For example, if a user says, "I've been feeling stressed lately, so I'd like to know how to relax," this audio data will be captured as is.

[0176] As a means of analysis within the server, audio data is converted into text data using speech analysis software. Here, a general speech recognition API is used for speech recognition technology. The converted text data is analyzed via a natural language processing engine to extract user needs and requests. The software used includes general-purpose language analysis tools and generative AI models.

[0177] Based on the analyzed results, a generation method is used to construct an action plan optimized for each individual user. In this process, the content of the plan is adjusted according to the emotions detected by the emotion analysis engine. For example, a user experiencing stress might be offered suggestions for meditation or relaxing music.

[0178] The action plan is communicated to the user via the device through various means. This communication can be in the form of a visual display or audio guide, and the user is informed that they can provide feedback on the plan's contents.

[0179] When users provide feedback, that information is returned to the server, and the feedback analysis system uses it to create future plans. This loop improves the user experience.

[0180] As a concrete example, by inputting the text prompt "Tell me how to relax" into the AI ​​model, it is possible to receive suggestions on relaxation techniques. In this way, the system of the present invention efficiently provides individualized and appropriate support to the user.

[0181] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0182] Step 1:

[0183] The device acquires audio from the user. Specifically, this involves capturing audio data using a microphone. The input is the user's raw voice data, which is then saved digitally.

[0184] Step 2:

[0185] The terminal transmits the acquired audio data to the server via the internet. A secure protocol is used for data transfer. The input is audio data, and the output is digital audio data transferred to the server.

[0186] Step 3:

[0187] The server converts the received audio data into text data using speech analysis software. Specifically, it uses speech recognition technology to analyze the audio signal and convert it into text. The input is digital audio data, and the output is the converted text data.

[0188] Step 4:

[0189] The server inputs the converted text data into a natural language processing engine to analyze the user's needs. It also uses an emotion analysis engine to identify the emotional state. The input is text data, and the analysis results include information on needs and emotional state.

[0190] Step 5:

[0191] The server uses a generative AI model based on the analysis results to generate a personalized action plan. Here, components are determined using pre-referenced databases and analysis results. The input is information on needs and emotional states, and the output is a customized action plan.

[0192] Step 6:

[0193] The terminal presents the generated action plan to the user. This involves communicating information using a display or audio output, providing visual or auditory feedback. The input is the action plan received from the server, and the output is the information presented to the user.

[0194] Step 7:

[0195] Users provide feedback on the action plan. This feedback is sent to the server via their device. The input consists of user ratings and requests, and the output is stored on the server as feedback information.

[0196] Step 8:

[0197] The server analyzes user feedback and updates the database to reflect it in future plan generation. Specifically, it analyzes the feedback to help adjust and improve the model. The input is user feedback, and the output is updated model and database information.

[0198] (Application Example 2)

[0199] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0200] In modern society, it is crucial for many people to appropriately address emotional challenges such as stress and loneliness that they experience in their daily lives. However, many households lack the immediate and appropriate support to meet their emotional needs. Therefore, there is a need for a system that can accurately understand each individual's emotional state and provide personalized solutions immediately based on that understanding.

[0201] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0202] In this invention, the server includes input means for acquiring voice or text data from the user, analysis means for analyzing the acquired data to identify the user's emotional state and needs and provide support suggestions in natural language, and generation means for generating a customized action plan based on the user's emotional state and needs. This makes it possible to immediately provide appropriate support in a home environment that is tailored to the user's emotional state.

[0203] "Input means" refers to a device or function that acquires voice or text data from a user.

[0204] "Analysis means" refers to a device or function that analyzes acquired data and identifies the user's emotional state and needs through natural language.

[0205] A "generation means" is a device or function that generates a customized action plan according to the emotional state and needs revealed through analysis.

[0206] "Output means" refers to a device or function that presents the generated action plan to the user and provides relevant support in their living environment.

[0207] "Support measures" refer to devices or functions that provide guidance policies to support the improvement of users' communication skills.

[0208] "Coordination means" refers to a device or function that performs the necessary coordination to facilitate cooperation with local organizations.

[0209] The system that realizes this invention is a home assistant that provides appropriate, emotion-based support in the user's living environment. The system utilizes a smart robot or terminal equipped with a microphone and includes the following key elements:

[0210] The server uses pyaudio to acquire audio data and the Google Speech Recognition API to convert the audio into text data in order to implement speech recognition technology. The converted text is then analyzed by a natural language processing engine, and sentiment analysis is performed using transformers during this process. Once the user's emotional state and needs are identified, the server uses machine learning models such as TENSORFLOW® to generate an action plan optimized for the user.

[0211] The device presents the generated action plan to the user via audio or visual means. If the user provides feedback, that data is sent back to the server and used to improve the quality of support in the future. This helps to alleviate daily stress and feelings of loneliness for elderly people and those living alone who require care at home.

[0212] For example, if an elderly user tells the device, "I'm feeling down today," the system identifies that emotion as "sadness" and suggests playing cheerful music or offering a warm drink. In this way, by understanding the user's emotions and responding appropriately, the system provides a sense of security.

[0213] The following are examples of prompt statements that utilize a generative AI model.

[0214] User input: "I'm feeling down today."

[0215] Prompt message: "The user may be feeling sad at the moment; please offer suggestions to cheer them up."

[0216] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0217] Step 1:

[0218] The device acquires the user's voice data. When the user speaks, it captures voice information through the microphone and collects it as audio data using the pyaudio library. The input is audio data, and the output is an audio signal that becomes available within the system.

[0219] Step 2:

[0220] The server converts the acquired audio data into text data using the Google Speech Recognition API. The input for this step is an audio signal, and the output after conversion is text data. Here, the content of the audio is recognized and replaced with text information.

[0221] Step 3:

[0222] The server uses the transformers library to analyze text data and perform natural language processing. The analysis identifies the user's needs and emotional state. This step takes string data as input and generates output containing information about the user's intentions and emotional state. Sentiment analysis extracts information such as whether a user's statement indicates "stress."

[0223] Step 4:

[0224] The server uses TensorFlow to generate a customized action plan based on the identified user's needs and emotional state. The input is information about emotions and needs, and the output is a list of appropriate actions. A machine learning model suggests the most effective course of action for the user.

[0225] Step 5:

[0226] The terminal presents the generated action plan to the user. This presentation is done either by displaying it on a screen or using an audio output device. In this step, the action plan is delivered to the user visually or audibly. The input is a list of action plans, and the output is a notification to the user.

[0227] Step 6:

[0228] Users provide feedback on the provided action plan and send it to the server via their device. The input is the user's feedback information, which the server receives and uses to improve future plans. The feedback information is stored on the server as output.

[0229] Step 7:

[0230] The server refines the action plan based on feedback received during the next user interaction. The input is past feedback data, and the output is an improved action plan. This step leverages past data to attempt further adaptation to the user.

[0231] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0232] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0233] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0234] [Second Embodiment]

[0235] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0236] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0237] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0238] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0239] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0240] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0241] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0242] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0243] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0244] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0245] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0246] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0247] The system of the present invention comprises an input device for handling user voice or text data, an analysis device for analyzing user data, a generation device for generating an action plan based on the analysis, an output device for presenting the plan, a support device for assisting dialogue skills, a legal support device for assisting legal procedures, and a collaboration device for coordinating with local communities. These devices work together to provide users with appropriate end-of-life support.

[0248] Program processing and specific examples

[0249] This indicates the user's intention to begin end-of-life planning.

[0250] The device captures the audio when the user states that they want to begin end-of-life planning, and sends it to the server for analysis.

[0251] The server converts this audio data into text and then performs natural language processing to identify the user's emotions and specific needs. For example, if a user says, "I want to write a will, but I don't know where to start," the analysis system detects that the user is feeling anxious.

[0252] Generating an optimized action plan

[0253] Based on the analysis, the server generates an action plan tailored to the user's situation. This plan details the necessary steps and procedures. For example, if the user wishes to create a will, the plan will specifically outline the legal requirements and procedures for doing so.

[0254] Presentation of action plans and communication support

[0255] The device presents the generated action plan to the user. If the user asks, "What exactly do I need to do?", the output device provides detailed steps. If the user needs assistance communicating with a parent, the support device advises on effective communication methods.

[0256] Support for legal procedures and community engagement

[0257] If the server determines that legal assistance is needed, it will provide legal information appropriate to the user's region through the legal assistance device. This may include details on necessary documents and procedures.

[0258] The connected device retrieves information on local end-of-life planning seminars and events based on the user's location and supports users who wish to participate. For example, it notifies users via their device that "there is a seminar being held at a nearby community center next week."

[0259] Thus, this invention provides services tailored to the specific needs and circumstances of users, creating a system that allows them to approach end-of-life planning with peace of mind.

[0260] The following describes the processing flow.

[0261] Step 1:

[0262] The device captures the user's voice expressing their end-of-life plans and collects the audio data. It verifies that the audio was captured correctly and prompts the user to re-enter any missing information.

[0263] Step 2:

[0264] The terminal instantly converts the captured audio data into text data and sends the data to the server.

[0265] Step 3:

[0266] The server analyzes text data using a natural language processing engine to identify the user's emotions and specific end-of-life planning needs. For example, it can determine if the user wishes to create a will or have their assets managed.

[0267] Step 4:

[0268] The server generates an appropriate action plan using a generator based on identified emotions and needs. This plan is customized according to the user's needs.

[0269] Step 5:

[0270] The device presents the generated action plan to the user in a viewable format. The user can review it via audio or text.

[0271] Step 6:

[0272] When a user asks about details of an action plan or a specific task, the device sends the question to the server to retrieve additional information.

[0273] Step 7:

[0274] The server provides relevant legal procedure information via a legal assistance device, based on the user's place of residence and needs. It presents the user with an overview of the necessary documents and procedures.

[0275] Step 8:

[0276] The server searches for information related to collaboration with local communities and retrieves information on end-of-life planning seminars and events using the linked device.

[0277] Step 9:

[0278] The device notifies the user of acquired local event information and confirms their willingness to participate. With the user's consent, it assists with registration for the event.

[0279] (Example 1)

[0280] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0281] In modern society, there is a need for an integrated support system to assist users in the end-of-life process. In particular, there is a demand for a system that has functions such as generating customized plans based on the user's psychological state, providing support for legal procedures, and facilitating cooperation with local communities. However, there is no consistent system that meets these requirements. The present invention aims to solve these problems.

[0282] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0283] In this invention, the server includes a receiving means for acquiring information from the user, an analyzing means for analyzing the acquired information to identify the user's psychological state and requirements, and a generating means for generating an individualized response plan based on the user's psychological state and requirements. As a result, the user can receive personalized support according to their own situation and needs. Also, by updating the generated plan, dynamic support corresponding to the user's changing situation can be provided.

[0284] The "receiving means" is a device or method for acquiring voice or text information transmitted from the user.

[0285] The "analyzing means" is a technology or process for analyzing the acquired data to identify the user's psychological state and requirements.

[0286] The "generating means" is a technology or process for creating an individualized response plan customized for the user based on the analysis result.

[0287] The "displaying means" is a device or technology for visualizing and providing the generated plan and information to the user.

[0288] The "supporting means" is a process having a function of providing various guidelines to improve the user's communication skills.

[0289] "Legal support measures" refer to methods or devices for providing information about legal procedures and for providing the legal support that users need.

[0290] "Collaboration tools" refer to technologies or processes that enable users to collaborate with local societies and activities in which they are involved.

[0291] The "update mechanism" is a process for updating the generated action plan in real time in response to additional user requests or changes in circumstances.

[0292] "Natural language processing technology" is an artificial intelligence technology that interprets human language through data analysis and extracts useful information.

[0293] A "generative AI model" is an artificial intelligence model used to achieve personalized responses and automation in user data analysis and plan generation.

[0294] This invention provides a comprehensive system to support users in their end-of-life planning. This system is designed with multiple devices and means working together.

[0295] First, the user inputs their request into the device via voice or text. At this stage, the voice data is converted into digital data. The device then sends the collected data to a server. The server uses natural language processing technology to analyze this data. Here, a generative AI model is used to extract the user's psychological state and specific needs.

[0296] Based on the analysis results, the server generates a personalized plan best suited to the user. This plan details the necessary procedures and steps, serving as a guide for the user's actions.

[0297] The generated action plan is presented to the user via the device. If the user has questions such as, "What exactly should I do?", the display will provide clear and detailed instructions.

[0298] Furthermore, if a user desires assistance with how to interact with others, the support system provides guidance on effective communication. If information on legal procedures is requested, the legal support system provides legal information relevant to the local area.

[0299] The collaborative method involves using users' local information to collect information on community activities and notifying users, thereby fostering cooperation with local communities.

[0300] A concrete example would be a situation where a user has a request such as, "I want to create a will, but I don't know where to start." In this case, an example of a prompt message to the generating AI model would be, "The user wants to know how to create a will. Please provide advice on where to begin."

[0301] This system provides users with flexible and personalized support tailored to their specific situation, playing a role in alleviating anxieties and doubts related to end-of-life planning.

[0302] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0303] Step 1:

[0304] The user inputs their end-of-life planning requests into the terminal via voice or text. These inputs are specific requests, such as "I want to create a will." The terminal converts the received voice data into a digital format and sends the resulting data to the service server.

[0305] Step 2:

[0306] The server receives the digital-formatted voice or text data sent from the terminal for analysis. Based on this input data, it uses natural language processing technology to perform processing to identify the user's mental state and needs. In this analysis, data processing such as converting voice data into text data is performed. Next, it proceeds with the preparation to input the identified mental state and needs into the generative AI model as output.

[0307] Step 3:

[0308] Based on the analysis results, the server uses the generative AI model to create an individualized response plan that meets the user's needs. At this time, the input is the user's state information obtained as the analysis result, and based on it, data operations are performed to output a customized action plan. This action plan includes the necessary procedures and steps and presents the actions that the user should take next.

[0309] Step 4:

[0310] The terminal receives the action plan sent from the server and presents it to the user through the display means. In this step, the received action plan is the input, and an operation of outputting visually or audibly through the user interface is performed.

[0311] Step 5:

[0312] While referring to the presented action plan, the user can request support for further details. For example, the user may input a question such as "How should I talk to my parents?" Based on this input, the support means generates and outputs effective communication advice. Through this operation, the user can learn practical conversation methods.

[0313] Step 6:

[0314] If deemed necessary, the server will provide localized legal information through legal assistance tools in response to the user's request regarding legal proceedings. This will include specific details on legally required documents and procedures, and output will be provided to assist the user in preparing their legal case.

[0315] Step 7:

[0316] The integration method involves obtaining information about end-of-life planning-related events taking place in the user's area based on their location information and notifying the user via their device. In this step, local information is input, and based on that information, information about related events is output and communicated to the user.

[0317] (Application Example 1)

[0318] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0319] When elderly people and their families are planning for the end of their lives or their future, they often feel anxious due to the complexity of information gathering and procedures. In particular, information on legal procedures and local support services is difficult to understand, and options for care are limited. Furthermore, there is a problem in that information on activities to improve quality of life is not effectively communicated. To solve these problems, a system is needed that flexibly provides users with appropriate and individualized information and support.

[0320] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0321] In this invention, the server includes a means for collecting and supporting participation in useful local activities based on the user's location, a means for providing a personalized life planning plan based on the analysis results and improving convenience, and an input means for acquiring the user's voice or text data. This makes it possible to provide personalized end-of-life planning and life planning support to the elderly and their families.

[0322] "Audio or text data" refers to information obtained from the user, including recorded audio and information expressed in text.

[0323] "Input means" refers to devices or interfaces for acquiring voice data or text data from users.

[0324] "Analysis means" refers to devices and algorithms that process input data to identify the user's emotional state and needs.

[0325] "Generation means" refers to devices and algorithms for creating an action plan optimized for the user based on the analysis results.

[0326] "Output means" refers to devices or interfaces used to present the generated action plan to the user.

[0327] "Support tools" refer to devices that provide guidance policies and support functions to improve users' conversational skills and communication abilities.

[0328] "Legal support measures" refer to devices that provide users with information and advice related to legal procedures they require.

[0329] "Collaboration tools" refer to devices that build relationships between users and their local communities, and provide useful information and opportunities for participation.

[0330] "Care support tools" refer to devices and functions that provide support for users' life planning and caregiving needs.

[0331] This invention is a system for providing care support, which uses voice or text data from the user to provide personalized support for end-of-life planning and life planning. The system consists of a device such as a smartphone or tablet and a server in the cloud.

[0332] The server uses speech recognition technology to convert the user's voice data into text, utilizing Google Cloud Speech-to-Text. This converted text data is then processed using the Google Cloud Natural Language API to analyze the user's emotional state and needs. Based on the analyzed information, an optimal action plan tailored to the user's situation is generated. This plan is then customized using a generative AI model in the cloud.

[0333] The generated action plan is presented to the user via their device, providing specific procedures, legal information, and information on local community activities. This helps users reduce anxiety and effectively prepare for end-of-life planning and long-term care.

[0334] For example, if a user is considering creating a will as part of "future preparations," their voice, such as "I want to write a will," is captured on the device and analyzed on the server. As a result, an action plan is generated that includes the legal requirements for creating a will and information on seminars held in the area, and this plan is presented to the user. Examples of prompts include, "What should I start with as preparations for the future?" or "Please tell me if there are any end-of-life planning events in my area."

[0335] Thus, the present invention provides a system that offers appropriate and personalized support to users, helping elderly people and their families to plan their lives with peace of mind.

[0336] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0337] Step 1:

[0338] The device acquires voice data from the user. The user inputs questions and wishes regarding end-of-life planning via voice through their smartphone or tablet. This acquired voice data is then sent directly to the server.

[0339] Step 2:

[0340] The server converts received audio data into text data using Google Cloud Speech-to-Text. The input is audio data, and the output is text data in which the audio has been converted into written information. This converted text data is then used for further analysis.

[0341] Step 3:

[0342] The server performs natural language processing on the transcribed data using the Google Cloud Natural Language API. In this step, it receives text data as input and extracts the user's emotional state and specific needs. The output is information indicating the user's emotional analysis results and specific needs.

[0343] Step 4:

[0344] The server generates an action plan using a generative AI model based on information obtained through natural language processing. The input here is the analysis results regarding the user's emotions and needs, and the output is a customized action plan for the user. This action plan can include specific legal procedures and information on local events.

[0345] Step 5:

[0346] The terminal displays an action plan received from the server to the user. The user can then review this plan and decide on specific actions to take regarding their end-of-life planning. The output is the action plan displayed on the terminal in a format that the user can directly view.

[0347] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0348] The system of the present invention comprises an input device and an analysis device for acquiring and analyzing user voice or text data. The input device captures the user's voice input, and the analysis device analyzes this data using a natural language processing engine and an emotion engine. The natural language processing engine understands the specific needs and requests spoken by the user, and the emotion engine identifies the user's emotional state from the context.

[0349] Program processing and specific examples

[0350] Recognizing the user's emotional state

[0351] When a device receives audio from a user, the server decodes the audio data and converts it into text data. For example, audio data such as "I've been stressed a lot lately and I'm worried" is converted into text.

[0352] The server uses a natural language processing engine to interpret user needs and simultaneously uses an emotion engine to recognize emotions such as "stress" and "worry."

[0353] Generating and adjusting action plans

[0354] The server generates a customized action plan based on analyzed needs, while also considering the user's emotional state.

[0355] The content of the action plan is adjusted according to the emotions recognized by the emotion engine. For example, a plan incorporating relaxation techniques will be suggested to a user who is feeling stressed.

[0356] Presenting plans to users and providing feedback.

[0357] The device presents the created action plan to the user visually or audibly. The user is also notified that they can ask additional questions or provide feedback on the content.

[0358] When a user provides feedback on a plan, that information is sent back to the server and used to generate future plans.

[0359] In this way, this system accurately understands the user's emotions and provides support plans accordingly, thereby reducing the anxiety and stress that users experience regarding end-of-life planning and providing comprehensive support.

[0360] The following describes the processing flow.

[0361] Step 1:

[0362] The user expresses their wishes and feelings regarding end-of-life planning in voice, and this voice data is captured by the device. The device receives this voice data, performs initial processing, and sends it to the server.

[0363] Step 2:

[0364] The server converts the received audio data into text data. The converted text is then input into the natural language processing engine and the emotion engine.

[0365] Step 3:

[0366] The server uses a natural language processing engine to extract user needs from the text. Simultaneously, an emotion engine analyzes the text to identify the user's emotional state. For example, it assesses levels of "anxiety," "stress," and "interest."

[0367] Step 4:

[0368] The server generates a personalized action plan based on the analyzed data. The action plan is adjusted according to the user's emotional state, as identified by the emotion engine. For example, a plan including relaxation techniques will be created for a user experiencing anxiety.

[0369] Step 5:

[0370] The device presents the generated action plan to the user. The user can choose whether to view it as text or receive it as an audio notification.

[0371] Step 6:

[0372] Users can review the presented action plan and provide feedback as needed. This feedback is sent to the server and used to improve or adjust the action plan.

[0373] Step 7:

[0374] The server stores user feedback in a database and considers that information when generating the next plan, preparing to provide more personalized support.

[0375] (Example 2)

[0376] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0377] In modern society, users often experience a variety of stresses and anxieties in their daily lives, creating a need for systems that provide appropriate support immediately. However, systems that can quickly and accurately recognize emotional states and specific needs and provide individualized support based on that information are still insufficient. Furthermore, more advanced analytical tools are needed to effectively utilize user feedback and provide customized plans for each user.

[0378] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0379] In this invention, the server includes acquisition means for acquiring voice and text data from the user, analysis means for analyzing the acquired data to identify the user's emotional state and needs, and feedback analysis means for receiving user feedback and utilizing that information to generate future plans. This makes it possible to quickly provide the user with an optimal action plan that is tailored to their emotions and needs, thereby improving the user's quality of life (QOL).

[0380] "Acquisition means" refers to a device or process for receiving voice or text data from a user and incorporating it into the system.

[0381] "Analysis means" refers to a device or method for analyzing acquired data to identify the user's emotional state and specific needs.

[0382] "Generation means" refers to an apparatus or process for creating a customized action plan tailored to the user based on the analysis results.

[0383] "Presentation means" refers to a device or method for presenting the generated action plan to the user visually or audibly.

[0384] A "feedback analysis means" is a device or system that receives feedback from users and utilizes that information to generate future plans.

[0385] "Conversion means" refers to a device or process that converts audio data into text data and prepares it for further analysis.

[0386] An "update mechanism" refers to a device or process for updating the action plan presented to the user in real time and making adjustments based on the user's situation and feedback.

[0387] This invention relates to a system that acquires and analyzes user voice or text data to provide a customized action plan tailored to the user's emotional state and specific needs. The system comprises acquisition means, analysis means, generation means, presentation means, feedback analysis means, and conversion means.

[0388] The server has a means of receiving audio from the user, and this audio data is converted into a digital format. The hardware used can be a device with a built-in microphone or an external microphone. For example, if a user says, "I've been feeling stressed lately, so I'd like to know how to relax," this audio data will be captured as is.

[0389] As a means of analysis within the server, audio data is converted into text data using speech analysis software. Here, a general speech recognition API is used for speech recognition technology. The converted text data is analyzed via a natural language processing engine to extract user needs and requests. The software used includes general-purpose language analysis tools and generative AI models.

[0390] Based on the analyzed results, a generation method is used to construct an action plan optimized for each individual user. In this process, the content of the plan is adjusted according to the emotions detected by the emotion analysis engine. For example, a user experiencing stress might be offered suggestions for meditation or relaxing music.

[0391] The action plan is communicated to the user via the device through various means. This communication can be in the form of a visual display or audio guide, and the user is informed that they can provide feedback on the plan's contents.

[0392] When users provide feedback, that information is returned to the server, and the feedback analysis system uses it to create future plans. This loop improves the user experience.

[0393] As a concrete example, by inputting the text prompt "Tell me how to relax" into the AI ​​model, it is possible to receive suggestions on relaxation techniques. In this way, the system of the present invention efficiently provides individualized and appropriate support to the user.

[0394] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0395] Step 1:

[0396] The device acquires audio from the user. Specifically, this involves capturing audio data using a microphone. The input is the user's raw voice data, which is then saved digitally.

[0397] Step 2:

[0398] The terminal transmits the acquired audio data to the server via the internet. A secure protocol is used for data transfer. The input is audio data, and the output is digital audio data transferred to the server.

[0399] Step 3:

[0400] The server converts the received audio data into text data using speech analysis software. Specifically, it uses speech recognition technology to analyze the audio signal and convert it into text. The input is digital audio data, and the output is the converted text data.

[0401] Step 4:

[0402] The server inputs the converted text data into a natural language processing engine to analyze the user's needs. It also uses an emotion analysis engine to identify the emotional state. The input is text data, and the analysis results include information on needs and emotional state.

[0403] Step 5:

[0404] The server uses a generative AI model based on the analysis results to generate a personalized action plan. Here, components are determined using pre-referenced databases and analysis results. The input is information on needs and emotional states, and the output is a customized action plan.

[0405] Step 6:

[0406] The terminal presents the generated action plan to the user. This involves communicating information using a display or audio output, providing visual or auditory feedback. The input is the action plan received from the server, and the output is the information presented to the user.

[0407] Step 7:

[0408] Users provide feedback on the action plan. This feedback is sent to the server via their device. The input consists of user ratings and requests, and the output is stored on the server as feedback information.

[0409] Step 8:

[0410] The server analyzes user feedback and updates the database to reflect it in future plan generation. Specifically, it analyzes the feedback to help adjust and improve the model. The input is user feedback, and the output is updated model and database information.

[0411] (Application Example 2)

[0412] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0413] In modern society, it is crucial for many people to appropriately address emotional challenges such as stress and loneliness that they experience in their daily lives. However, many households lack the immediate and appropriate support to meet their emotional needs. Therefore, there is a need for a system that can accurately understand each individual's emotional state and provide personalized solutions immediately based on that understanding.

[0414] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0415] In this invention, the server includes input means for acquiring voice or text data from the user, analysis means for analyzing the acquired data to identify the user's emotional state and needs and provide support suggestions in natural language, and generation means for generating a customized action plan based on the user's emotional state and needs. This makes it possible to immediately provide appropriate support in a home environment that is tailored to the user's emotional state.

[0416] "Input means" refers to a device or function that acquires voice or text data from a user.

[0417] "Analysis means" refers to a device or function that analyzes acquired data and identifies the user's emotional state and needs through natural language.

[0418] A "generation means" is a device or function that generates a customized action plan according to the emotional state and needs revealed through analysis.

[0419] "Output means" refers to a device or function that presents the generated action plan to the user and provides relevant support in their living environment.

[0420] "Support measures" refer to devices or functions that provide guidance policies to support the improvement of users' communication skills.

[0421] "Coordination means" refers to a device or function that performs the necessary coordination to facilitate cooperation with local organizations.

[0422] The system that realizes this invention is a home assistant that provides appropriate, emotion-based support in the user's living environment. The system utilizes a smart robot or terminal equipped with a microphone and includes the following key elements:

[0423] The server uses pyaudio to acquire audio data and the Google Speech Recognition API to convert the audio into text data. The converted text is then analyzed by a natural language processing engine, which performs sentiment analysis using transformers. Once the user's emotional state and needs are identified, the server uses machine learning models such as TensorFlow to generate an action plan optimized for the user.

[0424] The device presents the generated action plan to the user via audio or visual means. If the user provides feedback, that data is sent back to the server and used to improve the quality of support in the future. This helps to alleviate daily stress and feelings of loneliness for elderly people and those living alone who require care at home.

[0425] For example, if an elderly user tells the device, "I'm feeling down today," the system identifies that emotion as "sadness" and suggests playing cheerful music or offering a warm drink. In this way, by understanding the user's emotions and responding appropriately, the system provides a sense of security.

[0426] The following are examples of prompt statements that utilize a generative AI model.

[0427] User input: "I'm feeling down today."

[0428] Prompt message: "The user may be feeling sad at the moment; please offer suggestions to cheer them up."

[0429] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0430] Step 1:

[0431] The device acquires the user's voice data. When the user speaks, it captures voice information through the microphone and collects it as audio data using the pyaudio library. The input is audio data, and the output is an audio signal that becomes available within the system.

[0432] Step 2:

[0433] The server converts the acquired audio data into text data using the Google Speech Recognition API. The input for this step is an audio signal, and the output after conversion is text data. Here, the content of the audio is recognized and replaced with text information.

[0434] Step 3:

[0435] The server uses the transformers library to analyze text data and perform natural language processing. The analysis identifies the user's needs and emotional state. This step takes string data as input and generates output containing information about the user's intentions and emotional state. Sentiment analysis extracts information such as whether a user's statement indicates "stress."

[0436] Step 4:

[0437] The server uses TensorFlow to generate a customized action plan based on the identified user's needs and emotional state. The input is information about emotions and needs, and the output is a list of appropriate actions. A machine learning model suggests the most effective course of action for the user.

[0438] Step 5:

[0439] The terminal presents the generated action plan to the user. This presentation is done either by displaying it on a screen or using an audio output device. In this step, the action plan is delivered to the user visually or audibly. The input is a list of action plans, and the output is a notification to the user.

[0440] Step 6:

[0441] Users provide feedback on the provided action plan and send it to the server via their device. The input is the user's feedback information, which the server receives and uses to improve future plans. The feedback information is stored on the server as output.

[0442] Step 7:

[0443] The server refines the action plan based on feedback received during the next user interaction. The input is past feedback data, and the output is an improved action plan. This step leverages past data to attempt further adaptation to the user.

[0444] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0445] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0446] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0447] [Third Embodiment]

[0448] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0449] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0450] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0451] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0452] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0453] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0454] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0455] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0456] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0457] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0458] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0459] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0460] The system of the present invention comprises an input device for handling user voice or text data, an analysis device for analyzing user data, a generation device for generating an action plan based on the analysis, an output device for presenting the plan, a support device for assisting dialogue skills, a legal support device for assisting legal procedures, and a collaboration device for coordinating with local communities. These devices work together to provide users with appropriate end-of-life support.

[0461] Program processing and specific examples

[0462] This indicates the user's intention to begin end-of-life planning.

[0463] The device captures the audio when the user states that they want to begin end-of-life planning, and sends it to the server for analysis.

[0464] The server converts this audio data into text and then performs natural language processing to identify the user's emotions and specific needs. For example, if a user says, "I want to write a will, but I don't know where to start," the analysis system detects that the user is feeling anxious.

[0465] Generating an optimized action plan

[0466] Based on the analysis, the server generates an action plan tailored to the user's situation. This plan details the necessary steps and procedures. For example, if the user wishes to create a will, the plan will specifically outline the legal requirements and procedures for doing so.

[0467] Presentation of action plans and communication support

[0468] The device presents the generated action plan to the user. If the user asks, "What exactly do I need to do?", the output device provides detailed steps. If the user needs assistance communicating with a parent, the support device advises on effective communication methods.

[0469] Support for legal procedures and community engagement

[0470] If the server determines that legal assistance is needed, it will provide legal information appropriate to the user's region through the legal assistance device. This may include details on necessary documents and procedures.

[0471] The connected device retrieves information on local end-of-life planning seminars and events based on the user's location and supports users who wish to participate. For example, it notifies users via their device that "there is a seminar being held at a nearby community center next week."

[0472] Thus, this invention provides services tailored to the specific needs and circumstances of users, creating a system that allows them to approach end-of-life planning with peace of mind.

[0473] The following describes the processing flow.

[0474] Step 1:

[0475] The device captures the user's voice expressing their end-of-life plans and collects the audio data. It verifies that the audio was captured correctly and prompts the user to re-enter any missing information.

[0476] Step 2:

[0477] The terminal instantly converts the captured audio data into text data and sends the data to the server.

[0478] Step 3:

[0479] The server analyzes text data using a natural language processing engine to identify the user's emotions and specific end-of-life planning needs. For example, it can determine if the user wishes to create a will or have their assets managed.

[0480] Step 4:

[0481] The server generates an appropriate action plan using a generator based on identified emotions and needs. This plan is customized according to the user's needs.

[0482] Step 5:

[0483] The device presents the generated action plan to the user in a viewable format. The user can review it via audio or text.

[0484] Step 6:

[0485] When a user asks about details of an action plan or a specific task, the device sends the question to the server to retrieve additional information.

[0486] Step 7:

[0487] The server provides relevant legal procedure information via a legal assistance device, based on the user's place of residence and needs. It presents the user with an overview of the necessary documents and procedures.

[0488] Step 8:

[0489] The server searches for information related to collaboration with local communities and retrieves information on end-of-life planning seminars and events using the linked device.

[0490] Step 9:

[0491] The device notifies the user of acquired local event information and confirms their willingness to participate. With the user's consent, it assists with registration for the event.

[0492] (Example 1)

[0493] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0494] In modern society, a comprehensive support system is needed to assist users in their end-of-life planning process. In particular, a system is required that can generate customized plans based on the user's psychological state, provide support for legal procedures, and facilitate collaboration with local communities. However, a consistent system that meets these requirements does not yet exist. This invention aims to solve these problems.

[0495] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0496] In this invention, the server includes receiving means for acquiring information from the user, analyzing means for analyzing the acquired information to identify the user's psychological state and requests, and generating means for generating an individualized support plan based on the user's psychological state and requests. This enables the user to receive personalized support tailored to their own state and needs. Furthermore, by updating the generated plan, dynamic support can be provided that responds to changes in the user's situation.

[0497] "Receiving means" refers to a device or method for acquiring audio or text information transmitted by a user.

[0498] "Analysis means" refers to a technology or process that analyzes acquired data to identify the user's psychological state and needs.

[0499] "Generation method" refers to a technology or process for creating a customized, individualized response plan for the user based on the analysis results.

[0500] "Display means" refers to a device or technology for visualizing and providing generated plans and information to the user.

[0501] "Support measures" refer to processes that have the function of providing various guidance policies in order to improve users' communication skills.

[0502] "Legal support measures" refer to methods or devices for providing information about legal procedures and for providing the legal support that users need.

[0503] "Collaboration tools" refer to technologies or processes that enable users to collaborate with local societies and activities in which they are involved.

[0504] The "update mechanism" is a process for updating the generated action plan in real time in response to additional user requests or changes in circumstances.

[0505] "Natural language processing technology" is an artificial intelligence technology that interprets human language through data analysis and extracts useful information.

[0506] A "generative AI model" is an artificial intelligence model used to achieve personalized responses and automation in user data analysis and plan generation.

[0507] This invention provides a comprehensive system to support users in their end-of-life planning. This system is designed with multiple devices and means working together.

[0508] First, the user inputs their request into the device via voice or text. At this stage, the voice data is converted into digital data. The device then sends the collected data to a server. The server uses natural language processing technology to analyze this data. Here, a generative AI model is used to extract the user's psychological state and specific needs.

[0509] Based on the analysis results, the server generates a personalized plan best suited to the user. This plan details the necessary procedures and steps, serving as a guide for the user's actions.

[0510] The generated action plan is presented to the user via the device. If the user has questions such as, "What exactly should I do?", the display will provide clear and detailed instructions.

[0511] Furthermore, if a user desires assistance with how to interact with others, the support system provides guidance on effective communication. If information on legal procedures is requested, the legal support system provides legal information relevant to the local area.

[0512] The collaborative method involves using users' local information to collect information on community activities and notifying users, thereby fostering cooperation with local communities.

[0513] A concrete example would be a situation where a user has a request such as, "I want to create a will, but I don't know where to start." In this case, an example of a prompt message to the generating AI model would be, "The user wants to know how to create a will. Please provide advice on where to begin."

[0514] This system provides users with flexible and personalized support tailored to their specific situation, playing a role in alleviating anxieties and doubts related to end-of-life planning.

[0515] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0516] Step 1:

[0517] The user inputs their end-of-life planning requests into the terminal via voice or text. These inputs are specific requests, such as "I want to create a will." The terminal converts the received voice data into a digital format and sends the resulting data to the service server.

[0518] Step 2:

[0519] The server receives digital audio or text data sent from the terminal for analysis. Based on this input data, it uses natural language processing techniques to identify the user's psychological state and needs. This analysis involves data processing, such as converting audio data into text data. Next, it prepares to input the identified psychological state and needs as output into the generating AI model.

[0520] Step 3:

[0521] The server uses an AI model generated based on the analysis results to create a personalized response plan tailored to the user's needs. The input for this process is the user's state information obtained from the analysis results. Based on this, data calculations are performed to output a customized action plan. This action plan includes the necessary procedures and steps, outlining the next actions the user should take.

[0522] Step 4:

[0523] The terminal receives the action plan sent from the server and presents it to the user through a display device. In this step, the received action plan becomes input and is output visually or audibly through the user interface.

[0524] Step 5:

[0525] Users can refer to the presented action plan and then request further assistance with details. For example, they might input a question like, "How can I communicate with my parents?" Based on this input, the support system generates and outputs effective communication advice. This process allows users to learn practical communication methods.

[0526] Step 6:

[0527] If deemed necessary, the server will provide localized legal information through legal assistance tools in response to the user's request regarding legal proceedings. This will include specific details on legally required documents and procedures, and output will be provided to assist the user in preparing their legal case.

[0528] Step 7:

[0529] The integration method involves obtaining information about end-of-life planning-related events taking place in the user's area based on their location information and notifying the user via their device. In this step, local information is input, and based on that information, information about related events is output and communicated to the user.

[0530] (Application Example 1)

[0531] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0532] When elderly people and their families are planning for the end of their lives or their future, they often feel anxious due to the complexity of information gathering and procedures. In particular, information on legal procedures and local support services is difficult to understand, and options for care are limited. Furthermore, there is a problem in that information on activities to improve quality of life is not effectively communicated. To solve these problems, a system is needed that flexibly provides users with appropriate and individualized information and support.

[0533] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0534] In this invention, the server includes a means for collecting and supporting participation in useful local activities based on the user's location, a means for providing a personalized life planning plan based on the analysis results and improving convenience, and an input means for acquiring the user's voice or text data. This makes it possible to provide personalized end-of-life planning and life planning support to the elderly and their families.

[0535] "Audio or text data" refers to information obtained from the user, including recorded audio and information expressed in text.

[0536] "Input means" refers to devices or interfaces for acquiring voice data or text data from users.

[0537] "Analysis means" refers to devices and algorithms that process input data to identify the user's emotional state and needs.

[0538] "Generation means" refers to devices and algorithms for creating an action plan optimized for the user based on the analysis results.

[0539] "Output means" refers to devices or interfaces used to present the generated action plan to the user.

[0540] "Support tools" refer to devices that provide guidance policies and support functions to improve users' conversational skills and communication abilities.

[0541] "Legal support measures" refer to devices that provide users with information and advice related to legal procedures they require.

[0542] "Collaboration tools" refer to devices that build relationships between users and their local communities, and provide useful information and opportunities for participation.

[0543] "Care support tools" refer to devices and functions that provide support for users' life planning and caregiving needs.

[0544] This invention is a system for providing care support, which uses voice or text data from the user to provide personalized support for end-of-life planning and life planning. The system consists of a device such as a smartphone or tablet and a server in the cloud.

[0545] The server uses speech recognition technology to convert the user's voice data into text, utilizing Google Cloud Speech-to-Text. This converted text data is then processed using the Google Cloud Natural Language API to analyze the user's emotional state and needs. Based on the analyzed information, an optimal action plan tailored to the user's situation is generated. This plan is then customized using a generative AI model in the cloud.

[0546] The generated action plan is presented to the user via their device, providing specific procedures, legal information, and information on local community activities. This helps users reduce anxiety and effectively prepare for end-of-life planning and long-term care.

[0547] For example, if a user is considering creating a will as part of "future preparations," their voice, such as "I want to write a will," is captured on the device and analyzed on the server. As a result, an action plan is generated that includes the legal requirements for creating a will and information on seminars held in the area, and this plan is presented to the user. Examples of prompts include, "What should I start with as preparations for the future?" or "Please tell me if there are any end-of-life planning events in my area."

[0548] Thus, the present invention provides a system that offers appropriate and personalized support to users, helping elderly people and their families to plan their lives with peace of mind.

[0549] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0550] Step 1:

[0551] The device acquires voice data from the user. The user inputs questions and wishes regarding end-of-life planning via voice through their smartphone or tablet. This acquired voice data is then sent directly to the server.

[0552] Step 2:

[0553] The server converts received audio data into text data using Google Cloud Speech-to-Text. The input is audio data, and the output is text data in which the audio has been converted into written information. This converted text data is then used for further analysis.

[0554] Step 3:

[0555] The server performs natural language processing on the transcribed data using the Google Cloud Natural Language API. In this step, it receives text data as input and extracts the user's emotional state and specific needs. The output is information indicating the user's emotional analysis results and specific needs.

[0556] Step 4:

[0557] The server generates an action plan using a generative AI model based on information obtained through natural language processing. The input here is the analysis results regarding the user's emotions and needs, and the output is a customized action plan for the user. This action plan can include specific legal procedures and information on local events.

[0558] Step 5:

[0559] The terminal displays an action plan received from the server to the user. The user can then review this plan and decide on specific actions to take regarding their end-of-life planning. The output is the action plan displayed on the terminal in a format that the user can directly view.

[0560] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0561] The system of the present invention comprises an input device and an analysis device for acquiring and analyzing user voice or text data. The input device captures the user's voice input, and the analysis device analyzes this data using a natural language processing engine and an emotion engine. The natural language processing engine understands the specific needs and requests spoken by the user, and the emotion engine identifies the user's emotional state from the context.

[0562] Program processing and specific examples

[0563] Recognizing the user's emotional state

[0564] When a device receives audio from a user, the server decodes the audio data and converts it into text data. For example, audio data such as "I've been stressed a lot lately and I'm worried" is converted into text.

[0565] The server uses a natural language processing engine to interpret user needs and simultaneously uses an emotion engine to recognize emotions such as "stress" and "worry."

[0566] Generating and adjusting action plans

[0567] The server generates a customized action plan based on analyzed needs, while also considering the user's emotional state.

[0568] The content of the action plan is adjusted according to the emotions recognized by the emotion engine. For example, a plan incorporating relaxation techniques will be suggested to a user who is feeling stressed.

[0569] Presenting plans to users and providing feedback.

[0570] The device presents the created action plan to the user visually or audibly. The user is also notified that they can ask additional questions or provide feedback on the content.

[0571] When a user provides feedback on a plan, that information is sent back to the server and used to generate future plans.

[0572] In this way, this system accurately understands the user's emotions and provides support plans accordingly, thereby reducing the anxiety and stress that users experience regarding end-of-life planning and providing comprehensive support.

[0573] The following describes the processing flow.

[0574] Step 1:

[0575] The user expresses their wishes and feelings regarding end-of-life planning in voice, and this voice data is captured by the device. The device receives this voice data, performs initial processing, and sends it to the server.

[0576] Step 2:

[0577] The server converts the received audio data into text data. The converted text is then input into the natural language processing engine and the emotion engine.

[0578] Step 3:

[0579] The server uses a natural language processing engine to extract user needs from the text. Simultaneously, an emotion engine analyzes the text to identify the user's emotional state. For example, it assesses levels of "anxiety," "stress," and "interest."

[0580] Step 4:

[0581] The server generates a personalized action plan based on the analyzed data. The action plan is adjusted according to the user's emotional state, as identified by the emotion engine. For example, a plan including relaxation techniques will be created for a user experiencing anxiety.

[0582] Step 5:

[0583] The device presents the generated action plan to the user. The user can choose whether to view it as text or receive it as an audio notification.

[0584] Step 6:

[0585] Users can review the presented action plan and provide feedback as needed. This feedback is sent to the server and used to improve or adjust the action plan.

[0586] Step 7:

[0587] The server stores user feedback in a database and considers that information when generating the next plan, preparing to provide more personalized support.

[0588] (Example 2)

[0589] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0590] In modern society, users often experience a variety of stresses and anxieties in their daily lives, creating a need for systems that provide appropriate support immediately. However, systems that can quickly and accurately recognize emotional states and specific needs and provide individualized support based on that information are still insufficient. Furthermore, more advanced analytical tools are needed to effectively utilize user feedback and provide customized plans for each user.

[0591] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0592] In this invention, the server includes acquisition means for acquiring voice and text data from the user, analysis means for analyzing the acquired data to identify the user's emotional state and needs, and feedback analysis means for receiving user feedback and utilizing that information to generate future plans. This makes it possible to quickly provide the user with an optimal action plan that is tailored to their emotions and needs, thereby improving the user's quality of life (QOL).

[0593] "Acquisition means" refers to a device or process for receiving voice or text data from a user and incorporating it into the system.

[0594] "Analysis means" refers to a device or method for analyzing acquired data to identify the user's emotional state and specific needs.

[0595] "Generation means" refers to an apparatus or process for creating a customized action plan tailored to the user based on the analysis results.

[0596] "Presentation means" refers to a device or method for presenting the generated action plan to the user visually or audibly.

[0597] A "feedback analysis means" is a device or system that receives feedback from users and utilizes that information to generate future plans.

[0598] "Conversion means" refers to a device or process that converts audio data into text data and prepares it for further analysis.

[0599] An "update mechanism" refers to a device or process for updating the action plan presented to the user in real time and making adjustments based on the user's situation and feedback.

[0600] This invention relates to a system that acquires and analyzes user voice or text data to provide a customized action plan tailored to the user's emotional state and specific needs. The system comprises acquisition means, analysis means, generation means, presentation means, feedback analysis means, and conversion means.

[0601] The server has a means of receiving audio from the user, and this audio data is converted into a digital format. The hardware used can be a device with a built-in microphone or an external microphone. For example, if a user says, "I've been feeling stressed lately, so I'd like to know how to relax," this audio data will be captured as is.

[0602] As a means of analysis within the server, audio data is converted into text data using speech analysis software. Here, a general speech recognition API is used for speech recognition technology. The converted text data is analyzed via a natural language processing engine to extract user needs and requests. The software used includes general-purpose language analysis tools and generative AI models.

[0603] Based on the analyzed results, a generation method is used to construct an action plan optimized for each individual user. In this process, the content of the plan is adjusted according to the emotions detected by the emotion analysis engine. For example, a user experiencing stress might be offered suggestions for meditation or relaxing music.

[0604] The action plan is communicated to the user via the device through various means. This communication can be in the form of a visual display or audio guide, and the user is informed that they can provide feedback on the plan's contents.

[0605] When users provide feedback, that information is returned to the server, and the feedback analysis system uses it to create future plans. This loop improves the user experience.

[0606] As a concrete example, by inputting the text prompt "Tell me how to relax" into the AI ​​model, it is possible to receive suggestions on relaxation techniques. In this way, the system of the present invention efficiently provides individualized and appropriate support to the user.

[0607] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0608] Step 1:

[0609] The device acquires audio from the user. Specifically, this involves capturing audio data using a microphone. The input is the user's raw voice data, which is then saved digitally.

[0610] Step 2:

[0611] The terminal transmits the acquired audio data to the server via the internet. A secure protocol is used for data transfer. The input is audio data, and the output is digital audio data transferred to the server.

[0612] Step 3:

[0613] The server converts the received audio data into text data using speech analysis software. Specifically, it uses speech recognition technology to analyze the audio signal and convert it into text. The input is digital audio data, and the output is the converted text data.

[0614] Step 4:

[0615] The server inputs the converted text data into a natural language processing engine to analyze the user's needs. It also uses an emotion analysis engine to identify the emotional state. The input is text data, and the analysis results include information on needs and emotional state.

[0616] Step 5:

[0617] The server uses a generative AI model based on the analysis results to generate a personalized action plan. Here, components are determined using pre-referenced databases and analysis results. The input is information on needs and emotional states, and the output is a customized action plan.

[0618] Step 6:

[0619] The terminal presents the generated action plan to the user. This involves communicating information using a display or audio output, providing visual or auditory feedback. The input is the action plan received from the server, and the output is the information presented to the user.

[0620] Step 7:

[0621] Users provide feedback on the action plan. This feedback is sent to the server via their device. The input consists of user ratings and requests, and the output is stored on the server as feedback information.

[0622] Step 8:

[0623] The server analyzes user feedback and updates the database to reflect it in future plan generation. Specifically, it analyzes the feedback to help adjust and improve the model. The input is user feedback, and the output is updated model and database information.

[0624] (Application Example 2)

[0625] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0626] In modern society, it is crucial for many people to appropriately address emotional challenges such as stress and loneliness that they experience in their daily lives. However, many households lack the immediate and appropriate support to meet their emotional needs. Therefore, there is a need for a system that can accurately understand each individual's emotional state and provide personalized solutions immediately based on that understanding.

[0627] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0628] In this invention, the server includes input means for acquiring voice or text data from the user, analysis means for analyzing the acquired data to identify the user's emotional state and needs and provide support suggestions in natural language, and generation means for generating a customized action plan based on the user's emotional state and needs. This makes it possible to immediately provide appropriate support in a home environment that is tailored to the user's emotional state.

[0629] "Input means" refers to a device or function that acquires voice or text data from a user.

[0630] "Analysis means" refers to a device or function that analyzes acquired data and identifies the user's emotional state and needs through natural language.

[0631] A "generation means" is a device or function that generates a customized action plan according to the emotional state and needs revealed through analysis.

[0632] "Output means" refers to a device or function that presents the generated action plan to the user and provides relevant support in their living environment.

[0633] "Support measures" refer to devices or functions that provide guidance policies to support the improvement of users' communication skills.

[0634] "Coordination means" refers to a device or function that performs the necessary coordination to facilitate cooperation with local organizations.

[0635] The system that realizes this invention is a home assistant that provides appropriate, emotion-based support in the user's living environment. The system utilizes a smart robot or terminal equipped with a microphone and includes the following key elements:

[0636] The server uses pyaudio to acquire audio data and the Google Speech Recognition API to convert the audio into text data. The converted text is then analyzed by a natural language processing engine, which performs sentiment analysis using transformers. Once the user's emotional state and needs are identified, the server uses machine learning models such as TensorFlow to generate an action plan optimized for the user.

[0637] The device presents the generated action plan to the user via audio or visual means. If the user provides feedback, that data is sent back to the server and used to improve the quality of support in the future. This helps to alleviate daily stress and feelings of loneliness for elderly people and those living alone who require care at home.

[0638] For example, if an elderly user tells the device, "I'm feeling down today," the system identifies that emotion as "sadness" and suggests playing cheerful music or offering a warm drink. In this way, by understanding the user's emotions and responding appropriately, the system provides a sense of security.

[0639] The following are examples of prompt statements that utilize a generative AI model.

[0640] User input: "I'm feeling down today."

[0641] Prompt message: "The user may be feeling sad at the moment; please offer suggestions to cheer them up."

[0642] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0643] Step 1:

[0644] The device acquires the user's voice data. When the user speaks, it captures voice information through the microphone and collects it as audio data using the pyaudio library. The input is audio data, and the output is an audio signal that becomes available within the system.

[0645] Step 2:

[0646] The server converts the acquired audio data into text data using the Google Speech Recognition API. The input for this step is an audio signal, and the output after conversion is text data. Here, the content of the audio is recognized and replaced with text information.

[0647] Step 3:

[0648] The server uses the transformers library to analyze text data and perform natural language processing. The analysis identifies the user's needs and emotional state. This step takes string data as input and generates output containing information about the user's intentions and emotional state. Sentiment analysis extracts information such as whether a user's statement indicates "stress."

[0649] Step 4:

[0650] The server uses TensorFlow to generate a customized action plan based on the identified user's needs and emotional state. The input is information about emotions and needs, and the output is a list of appropriate actions. A machine learning model suggests the most effective course of action for the user.

[0651] Step 5:

[0652] The terminal presents the generated action plan to the user. This presentation is done either by displaying it on a screen or using an audio output device. In this step, the action plan is delivered to the user visually or audibly. The input is a list of action plans, and the output is a notification to the user.

[0653] Step 6:

[0654] Users provide feedback on the provided action plan and send it to the server via their device. The input is the user's feedback information, which the server receives and uses to improve future plans. The feedback information is stored on the server as output.

[0655] Step 7:

[0656] The server refines the action plan based on feedback received during the next user interaction. The input is past feedback data, and the output is an improved action plan. This step leverages past data to attempt further adaptation to the user.

[0657] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0658] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0659] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0660] [Fourth Embodiment]

[0661] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0662] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0663] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0664] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0665] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0666] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0667] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0668] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0669] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0670] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0671] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0672] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0673] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0674] The system of the present invention comprises an input device for handling user voice or text data, an analysis device for analyzing user data, a generation device for generating an action plan based on the analysis, an output device for presenting the plan, a support device for assisting dialogue skills, a legal support device for assisting legal procedures, and a collaboration device for coordinating with local communities. These devices work together to provide users with appropriate end-of-life support.

[0675] Program processing and specific examples

[0676] This indicates the user's intention to begin end-of-life planning.

[0677] The device captures the audio when the user states that they want to begin end-of-life planning, and sends it to the server for analysis.

[0678] The server converts this audio data into text and then performs natural language processing to identify the user's emotions and specific needs. For example, if a user says, "I want to write a will, but I don't know where to start," the analysis system detects that the user is feeling anxious.

[0679] Generating an optimized action plan

[0680] Based on the analysis, the server generates an action plan tailored to the user's situation. This plan details the necessary steps and procedures. For example, if the user wishes to create a will, the plan will specifically outline the legal requirements and procedures for doing so.

[0681] Presentation of action plans and communication support

[0682] The device presents the generated action plan to the user. If the user asks, "What exactly do I need to do?", the output device provides detailed steps. If the user needs assistance communicating with a parent, the support device advises on effective communication methods.

[0683] Support for legal procedures and community engagement

[0684] If the server determines that legal assistance is needed, it will provide legal information appropriate to the user's region through the legal assistance device. This may include details on necessary documents and procedures.

[0685] The connected device retrieves information on local end-of-life planning seminars and events based on the user's location and supports users who wish to participate. For example, it notifies users via their device that "there is a seminar being held at a nearby community center next week."

[0686] Thus, this invention provides services tailored to the specific needs and circumstances of users, creating a system that allows them to approach end-of-life planning with peace of mind.

[0687] The following describes the processing flow.

[0688] Step 1:

[0689] The device captures the user's voice expressing their end-of-life plans and collects the audio data. It verifies that the audio was captured correctly and prompts the user to re-enter any missing information.

[0690] Step 2:

[0691] The terminal instantly converts the captured audio data into text data and sends the data to the server.

[0692] Step 3:

[0693] The server analyzes text data using a natural language processing engine to identify the user's emotions and specific end-of-life planning needs. For example, it can determine if the user wishes to create a will or have their assets managed.

[0694] Step 4:

[0695] The server generates an appropriate action plan using a generator based on identified emotions and needs. This plan is customized according to the user's needs.

[0696] Step 5:

[0697] The device presents the generated action plan to the user in a viewable format. The user can review it via audio or text.

[0698] Step 6:

[0699] When a user asks about details of an action plan or a specific task, the device sends the question to the server to retrieve additional information.

[0700] Step 7:

[0701] The server provides relevant legal procedure information via a legal assistance device, based on the user's place of residence and needs. It presents the user with an overview of the necessary documents and procedures.

[0702] Step 8:

[0703] The server searches for information related to collaboration with local communities and retrieves information on end-of-life planning seminars and events using the linked device.

[0704] Step 9:

[0705] The device notifies the user of acquired local event information and confirms their willingness to participate. With the user's consent, it assists with registration for the event.

[0706] (Example 1)

[0707] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0708] In modern society, a comprehensive support system is needed to assist users in their end-of-life planning process. In particular, a system is required that can generate customized plans based on the user's psychological state, provide support for legal procedures, and facilitate collaboration with local communities. However, a consistent system that meets these requirements does not yet exist. This invention aims to solve these problems.

[0709] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0710] In this invention, the server includes receiving means for acquiring information from the user, analyzing means for analyzing the acquired information to identify the user's psychological state and requests, and generating means for generating an individualized support plan based on the user's psychological state and requests. This enables the user to receive personalized support tailored to their own state and needs. Furthermore, by updating the generated plan, dynamic support can be provided that responds to changes in the user's situation.

[0711] "Receiving means" refers to a device or method for acquiring audio or text information transmitted by a user.

[0712] "Analysis means" refers to a technology or process that analyzes acquired data to identify the user's psychological state and needs.

[0713] "Generation method" refers to a technology or process for creating a customized, individualized response plan for the user based on the analysis results.

[0714] "Display means" refers to a device or technology for visualizing and providing generated plans and information to the user.

[0715] "Support measures" refer to processes that have the function of providing various guidance policies in order to improve users' communication skills.

[0716] "Legal support measures" refer to methods or devices for providing information about legal procedures and for providing the legal support that users need.

[0717] "Collaboration tools" refer to technologies or processes that enable users to collaborate with local societies and activities in which they are involved.

[0718] The "update mechanism" is a process for updating the generated action plan in real time in response to additional user requests or changes in circumstances.

[0719] "Natural language processing technology" is an artificial intelligence technology that interprets human language through data analysis and extracts useful information.

[0720] A "generative AI model" is an artificial intelligence model used to achieve personalized responses and automation in user data analysis and plan generation.

[0721] This invention provides a comprehensive system to support users in their end-of-life planning. This system is designed with multiple devices and means working together.

[0722] First, the user inputs their request into the device via voice or text. At this stage, the voice data is converted into digital data. The device then sends the collected data to a server. The server uses natural language processing technology to analyze this data. Here, a generative AI model is used to extract the user's psychological state and specific needs.

[0723] Based on the analysis results, the server generates a personalized plan best suited to the user. This plan details the necessary procedures and steps, serving as a guide for the user's actions.

[0724] The generated action plan is presented to the user via the device. If the user has questions such as, "What exactly should I do?", the display will provide clear and detailed instructions.

[0725] Furthermore, if a user desires assistance with how to interact with others, the support system provides guidance on effective communication. If information on legal procedures is requested, the legal support system provides legal information relevant to the local area.

[0726] The collaborative method involves using users' local information to collect information on community activities and notifying users, thereby fostering cooperation with local communities.

[0727] A concrete example would be a situation where a user has a request such as, "I want to create a will, but I don't know where to start." In this case, an example of a prompt message to the generating AI model would be, "The user wants to know how to create a will. Please provide advice on where to begin."

[0728] This system provides users with flexible and personalized support tailored to their specific situation, playing a role in alleviating anxieties and doubts related to end-of-life planning.

[0729] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0730] Step 1:

[0731] The user inputs their end-of-life planning requests into the terminal via voice or text. These inputs are specific requests, such as "I want to create a will." The terminal converts the received voice data into a digital format and sends the resulting data to the service server.

[0732] Step 2:

[0733] The server receives digital audio or text data sent from the terminal for analysis. Based on this input data, it uses natural language processing techniques to identify the user's psychological state and needs. This analysis involves data processing, such as converting audio data into text data. Next, it prepares to input the identified psychological state and needs as output into the generating AI model.

[0734] Step 3:

[0735] The server uses an AI model generated based on the analysis results to create a personalized response plan tailored to the user's needs. The input for this process is the user's state information obtained from the analysis results. Based on this, data calculations are performed to output a customized action plan. This action plan includes the necessary procedures and steps, outlining the next actions the user should take.

[0736] Step 4:

[0737] The terminal receives the action plan sent from the server and presents it to the user through a display device. In this step, the received action plan becomes input and is output visually or audibly through the user interface.

[0738] Step 5:

[0739] Users can refer to the presented action plan and then request further assistance with details. For example, they might input a question like, "How can I communicate with my parents?" Based on this input, the support system generates and outputs effective communication advice. This process allows users to learn practical communication methods.

[0740] Step 6:

[0741] If deemed necessary, the server will provide localized legal information through legal assistance tools in response to the user's request regarding legal proceedings. This will include specific details on legally required documents and procedures, and output will be provided to assist the user in preparing their legal case.

[0742] Step 7:

[0743] The integration method involves obtaining information about end-of-life planning-related events taking place in the user's area based on their location information and notifying the user via their device. In this step, local information is input, and based on that information, information about related events is output and communicated to the user.

[0744] (Application Example 1)

[0745] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0746] When elderly people and their families are planning for the end of their lives or their future, they often feel anxious due to the complexity of information gathering and procedures. In particular, information on legal procedures and local support services is difficult to understand, and options for care are limited. Furthermore, there is a problem in that information on activities to improve quality of life is not effectively communicated. To solve these problems, a system is needed that flexibly provides users with appropriate and individualized information and support.

[0747] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0748] In this invention, the server includes a means for collecting and supporting participation in useful local activities based on the user's location, a means for providing a personalized life planning plan based on the analysis results and improving convenience, and an input means for acquiring the user's voice or text data. This makes it possible to provide personalized end-of-life planning and life planning support to the elderly and their families.

[0749] "Audio or text data" refers to information obtained from the user, including recorded audio and information expressed in text.

[0750] "Input means" refers to devices or interfaces for acquiring voice data or text data from users.

[0751] "Analysis means" refers to devices and algorithms that process input data to identify the user's emotional state and needs.

[0752] "Generation means" refers to devices and algorithms for creating an action plan optimized for the user based on the analysis results.

[0753] "Output means" refers to devices or interfaces used to present the generated action plan to the user.

[0754] "Support tools" refer to devices that provide guidance policies and support functions to improve users' conversational skills and communication abilities.

[0755] "Legal support measures" refer to devices that provide users with information and advice related to legal procedures they require.

[0756] "Collaboration tools" refer to devices that build relationships between users and their local communities, and provide useful information and opportunities for participation.

[0757] "Care support tools" refer to devices and functions that provide support for users' life planning and caregiving needs.

[0758] This invention is a system for providing care support, which uses voice or text data from the user to provide personalized support for end-of-life planning and life planning. The system consists of a device such as a smartphone or tablet and a server in the cloud.

[0759] The server uses speech recognition technology to convert the user's voice data into text, utilizing Google Cloud Speech-to-Text. This converted text data is then processed using the Google Cloud Natural Language API to analyze the user's emotional state and needs. Based on the analyzed information, an optimal action plan tailored to the user's situation is generated. This plan is then customized using a generative AI model in the cloud.

[0760] The generated action plan is presented to the user via their device, providing specific procedures, legal information, and information on local community activities. This helps users reduce anxiety and effectively prepare for end-of-life planning and long-term care.

[0761] For example, if a user is considering creating a will as part of "future preparations," their voice, such as "I want to write a will," is captured on the device and analyzed on the server. As a result, an action plan is generated that includes the legal requirements for creating a will and information on seminars held in the area, and this plan is presented to the user. Examples of prompts include, "What should I start with as preparations for the future?" or "Please tell me if there are any end-of-life planning events in my area."

[0762] Thus, the present invention provides a system that offers appropriate and personalized support to users, helping elderly people and their families to plan their lives with peace of mind.

[0763] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0764] Step 1:

[0765] The device acquires voice data from the user. The user inputs questions and wishes regarding end-of-life planning via voice through their smartphone or tablet. This acquired voice data is then sent directly to the server.

[0766] Step 2:

[0767] The server converts received audio data into text data using Google Cloud Speech-to-Text. The input is audio data, and the output is text data in which the audio has been converted into written information. This converted text data is then used for further analysis.

[0768] Step 3:

[0769] The server performs natural language processing on the transcribed data using the Google Cloud Natural Language API. In this step, it receives text data as input and extracts the user's emotional state and specific needs. The output is information indicating the user's emotional analysis results and specific needs.

[0770] Step 4:

[0771] The server generates an action plan using a generative AI model based on information obtained through natural language processing. The input here is the analysis results regarding the user's emotions and needs, and the output is a customized action plan for the user. This action plan can include specific legal procedures and information on local events.

[0772] Step 5:

[0773] The terminal displays an action plan received from the server to the user. The user can then review this plan and decide on specific actions to take regarding their end-of-life planning. The output is the action plan displayed on the terminal in a format that the user can directly view.

[0774] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0775] The system of the present invention comprises an input device and an analysis device for acquiring and analyzing user voice or text data. The input device captures the user's voice input, and the analysis device analyzes this data using a natural language processing engine and an emotion engine. The natural language processing engine understands the specific needs and requests spoken by the user, and the emotion engine identifies the user's emotional state from the context.

[0776] Program processing and specific examples

[0777] Recognizing the user's emotional state

[0778] When a device receives audio from a user, the server decodes the audio data and converts it into text data. For example, audio data such as "I've been stressed a lot lately and I'm worried" is converted into text.

[0779] The server uses a natural language processing engine to interpret user needs and simultaneously uses an emotion engine to recognize emotions such as "stress" and "worry."

[0780] Generating and adjusting action plans

[0781] The server generates a customized action plan based on analyzed needs, while also considering the user's emotional state.

[0782] The content of the action plan is adjusted according to the emotions recognized by the emotion engine. For example, a plan incorporating relaxation techniques will be suggested to a user who is feeling stressed.

[0783] Presenting plans to users and providing feedback.

[0784] The device presents the created action plan to the user visually or audibly. The user is also notified that they can ask additional questions or provide feedback on the content.

[0785] When a user provides feedback on a plan, that information is sent back to the server and used to generate future plans.

[0786] In this way, this system accurately understands the user's emotions and provides support plans accordingly, thereby reducing the anxiety and stress that users experience regarding end-of-life planning and providing comprehensive support.

[0787] The following describes the processing flow.

[0788] Step 1:

[0789] The user expresses their wishes and feelings regarding end-of-life planning in voice, and this voice data is captured by the device. The device receives this voice data, performs initial processing, and sends it to the server.

[0790] Step 2:

[0791] The server converts the received audio data into text data. The converted text is then input into the natural language processing engine and the emotion engine.

[0792] Step 3:

[0793] The server uses a natural language processing engine to extract user needs from the text. Simultaneously, an emotion engine analyzes the text to identify the user's emotional state. For example, it assesses levels of "anxiety," "stress," and "interest."

[0794] Step 4:

[0795] The server generates a personalized action plan based on the analyzed data. The action plan is adjusted according to the user's emotional state, as identified by the emotion engine. For example, a plan including relaxation techniques will be created for a user experiencing anxiety.

[0796] Step 5:

[0797] The device presents the generated action plan to the user. The user can choose whether to view it as text or receive it as an audio notification.

[0798] Step 6:

[0799] Users can review the presented action plan and provide feedback as needed. This feedback is sent to the server and used to improve or adjust the action plan.

[0800] Step 7:

[0801] The server stores user feedback in a database and considers that information when generating the next plan, preparing to provide more personalized support.

[0802] (Example 2)

[0803] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0804] In modern society, users often experience a variety of stresses and anxieties in their daily lives, creating a need for systems that provide appropriate support immediately. However, systems that can quickly and accurately recognize emotional states and specific needs and provide individualized support based on that information are still insufficient. Furthermore, more advanced analytical tools are needed to effectively utilize user feedback and provide customized plans for each user.

[0805] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0806] In this invention, the server includes acquisition means for acquiring voice and text data from the user, analysis means for analyzing the acquired data to identify the user's emotional state and needs, and feedback analysis means for receiving user feedback and utilizing that information to generate future plans. This makes it possible to quickly provide the user with an optimal action plan that is tailored to their emotions and needs, thereby improving the user's quality of life (QOL).

[0807] "Acquisition means" refers to a device or process for receiving voice or text data from a user and incorporating it into the system.

[0808] "Analysis means" refers to a device or method for analyzing acquired data to identify the user's emotional state and specific needs.

[0809] "Generation means" refers to an apparatus or process for creating a customized action plan tailored to the user based on the analysis results.

[0810] "Presentation means" refers to a device or method for presenting the generated action plan to the user visually or audibly.

[0811] A "feedback analysis means" is a device or system that receives feedback from users and utilizes that information to generate future plans.

[0812] "Conversion means" refers to a device or process that converts audio data into text data and prepares it for further analysis.

[0813] An "update mechanism" refers to a device or process for updating the action plan presented to the user in real time and making adjustments based on the user's situation and feedback.

[0814] This invention relates to a system that acquires and analyzes user voice or text data to provide a customized action plan tailored to the user's emotional state and specific needs. The system comprises acquisition means, analysis means, generation means, presentation means, feedback analysis means, and conversion means.

[0815] The server has a means of receiving audio from the user, and this audio data is converted into a digital format. The hardware used can be a device with a built-in microphone or an external microphone. For example, if a user says, "I've been feeling stressed lately, so I'd like to know how to relax," this audio data will be captured as is.

[0816] As a means of analysis within the server, audio data is converted into text data using speech analysis software. Here, a general speech recognition API is used for speech recognition technology. The converted text data is analyzed via a natural language processing engine to extract user needs and requests. The software used includes general-purpose language analysis tools and generative AI models.

[0817] Based on the analyzed results, a generation method is used to construct an action plan optimized for each individual user. In this process, the content of the plan is adjusted according to the emotions detected by the emotion analysis engine. For example, a user experiencing stress might be offered suggestions for meditation or relaxing music.

[0818] The action plan is communicated to the user via the device through various means. This communication can be in the form of a visual display or audio guide, and the user is informed that they can provide feedback on the plan's contents.

[0819] When users provide feedback, that information is returned to the server, and the feedback analysis system uses it to create future plans. This loop improves the user experience.

[0820] As a concrete example, by inputting the text prompt "Tell me how to relax" into the AI ​​model, it is possible to receive suggestions on relaxation techniques. In this way, the system of the present invention efficiently provides individualized and appropriate support to the user.

[0821] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0822] Step 1:

[0823] The device acquires audio from the user. Specifically, this involves capturing audio data using a microphone. The input is the user's raw voice data, which is then saved digitally.

[0824] Step 2:

[0825] The terminal transmits the acquired audio data to the server via the internet. A secure protocol is used for data transfer. The input is audio data, and the output is digital audio data transferred to the server.

[0826] Step 3:

[0827] The server converts the received audio data into text data using speech analysis software. Specifically, it uses speech recognition technology to analyze the audio signal and convert it into text. The input is digital audio data, and the output is the converted text data.

[0828] Step 4:

[0829] The server inputs the converted text data into a natural language processing engine to analyze the user's needs. It also uses an emotion analysis engine to identify the emotional state. The input is text data, and the analysis results include information on needs and emotional state.

[0830] Step 5:

[0831] The server uses a generative AI model based on the analysis results to generate a personalized action plan. Here, components are determined using pre-referenced databases and analysis results. The input is information on needs and emotional states, and the output is a customized action plan.

[0832] Step 6:

[0833] The terminal presents the generated action plan to the user. This involves communicating information using a display or audio output, providing visual or auditory feedback. The input is the action plan received from the server, and the output is the information presented to the user.

[0834] Step 7:

[0835] Users provide feedback on the action plan. This feedback is sent to the server via their device. The input consists of user ratings and requests, and the output is stored on the server as feedback information.

[0836] Step 8:

[0837] The server analyzes user feedback and updates the database to reflect it in future plan generation. Specifically, it analyzes the feedback to help adjust and improve the model. The input is user feedback, and the output is updated model and database information.

[0838] (Application Example 2)

[0839] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0840] In modern society, it is crucial for many people to appropriately address emotional challenges such as stress and loneliness that they experience in their daily lives. However, many households lack the immediate and appropriate support to meet their emotional needs. Therefore, there is a need for a system that can accurately understand each individual's emotional state and provide personalized solutions immediately based on that understanding.

[0841] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0842] In this invention, the server includes input means for acquiring voice or text data from the user, analysis means for analyzing the acquired data to identify the user's emotional state and needs and provide support suggestions in natural language, and generation means for generating a customized action plan based on the user's emotional state and needs. This makes it possible to immediately provide appropriate support in a home environment that is tailored to the user's emotional state.

[0843] "Input means" refers to a device or function that acquires voice or text data from a user.

[0844] "Analysis means" refers to a device or function that analyzes acquired data and identifies the user's emotional state and needs through natural language.

[0845] A "generation means" is a device or function that generates a customized action plan according to the emotional state and needs revealed through analysis.

[0846] "Output means" refers to a device or function that presents the generated action plan to the user and provides relevant support in their living environment.

[0847] "Support measures" refer to devices or functions that provide guidance policies to support the improvement of users' communication skills.

[0848] "Coordination means" refers to a device or function that performs the necessary coordination to facilitate cooperation with local organizations.

[0849] The system that realizes this invention is a home assistant that provides appropriate, emotion-based support in the user's living environment. The system utilizes a smart robot or terminal equipped with a microphone and includes the following key elements:

[0850] The server uses pyaudio to acquire audio data and the Google Speech Recognition API to convert the audio into text data. The converted text is then analyzed by a natural language processing engine, which performs sentiment analysis using transformers. Once the user's emotional state and needs are identified, the server uses machine learning models such as TensorFlow to generate an action plan optimized for the user.

[0851] The device presents the generated action plan to the user via audio or visual means. If the user provides feedback, that data is sent back to the server and used to improve the quality of support in the future. This helps to alleviate daily stress and feelings of loneliness for elderly people and those living alone who require care at home.

[0852] For example, if an elderly user tells the device, "I'm feeling down today," the system identifies that emotion as "sadness" and suggests playing cheerful music or offering a warm drink. In this way, by understanding the user's emotions and responding appropriately, the system provides a sense of security.

[0853] The following are examples of prompt statements that utilize a generative AI model.

[0854] User input: "I'm feeling down today."

[0855] Prompt message: "The user may be feeling sad at the moment; please offer suggestions to cheer them up."

[0856] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0857] Step 1:

[0858] The device acquires the user's voice data. When the user speaks, it captures voice information through the microphone and collects it as audio data using the pyaudio library. The input is audio data, and the output is an audio signal that becomes available within the system.

[0859] Step 2:

[0860] The server converts the acquired audio data into text data using the Google Speech Recognition API. The input for this step is an audio signal, and the output after conversion is text data. Here, the content of the audio is recognized and replaced with text information.

[0861] Step 3:

[0862] The server uses the transformers library to analyze text data and perform natural language processing. The analysis identifies the user's needs and emotional state. This step takes string data as input and generates output containing information about the user's intentions and emotional state. Sentiment analysis extracts information such as whether a user's statement indicates "stress."

[0863] Step 4:

[0864] The server uses TensorFlow to generate a customized action plan based on the identified user's needs and emotional state. The input is information about emotions and needs, and the output is a list of appropriate actions. A machine learning model suggests the most effective course of action for the user.

[0865] Step 5:

[0866] The terminal presents the generated action plan to the user. This presentation is done either by displaying it on a screen or using an audio output device. In this step, the action plan is delivered to the user visually or audibly. The input is a list of action plans, and the output is a notification to the user.

[0867] Step 6:

[0868] Users provide feedback on the provided action plan and send it to the server via their device. The input is the user's feedback information, which the server receives and uses to improve future plans. The feedback information is stored on the server as output.

[0869] Step 7:

[0870] The server refines the action plan based on feedback received during the next user interaction. The input is past feedback data, and the output is an improved action plan. This step leverages past data to attempt further adaptation to the user.

[0871] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0872] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0873] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0874] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0875] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0876] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0877] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0878] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0879] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0880] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0881] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0882] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0883] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0884] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0885] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0886] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0887] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0888] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0889] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0890] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0891] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0892] The following is further disclosed regarding the embodiments described above.

[0893] (Claim 1)

[0894] An input device that acquires voice or text data from the user,

[0895] An analysis device that analyzes acquired data to identify the user's emotional state and needs,

[0896] A generator that generates a customized action plan based on the user's emotional state and needs,

[0897] An output device that presents the generated action plan to the user,

[0898] A support device that provides communication guidelines to help improve users' conversational skills,

[0899] A legal support device that provides information on legal procedures,

[0900] A collaborative device that facilitates connection between users and related local communities,

[0901] A system that includes this.

[0902] (Claim 2)

[0903] The system according to claim 1, further comprising an analysis means that uses a natural language processing engine to identify the emotional state of a user.

[0904] (Claim 3)

[0905] The system according to claim 1, comprising an update means for updating the action plan presented to the user in real time.

[0906] "Example 1"

[0907] (Claim 1)

[0908] A means of receiving information from the user,

[0909] An analytical means that analyzes acquired information to identify the user's psychological state and requests,

[0910] A generation means for generating individualized response plans based on the user's psychological state and requests,

[0911] A display means for presenting the generated individualized plan to the user,

[0912] A support tool that provides guidance policies to help improve users' communication skills,

[0913] Legal support measures that provide information on legal procedures,

[0914] A means of cooperation to foster collaboration between users and the local communities they interact with,

[0915] A processing means that converts speech data into text data based on natural language processing technology used in the analysis means,

[0916] A means for updating the generated action plan based on additional user requests,

[0917] A system that includes this.

[0918] (Claim 2)

[0919] The system according to claim 1, wherein a generative AI model is used when implementing natural language processing technology in the analysis means.

[0920] (Claim 3)

[0921] The system according to claim 1, further comprising a cooperative means for acquiring information on local activities based on the user's location information and guiding the user accordingly.

[0922] "Application Example 1"

[0923] (Claim 1)

[0924] An input means for acquiring user voice or text data,

[0925] An analytical method that analyzes acquired data to identify the user's emotional state and needs,

[0926] A generation means for generating a customized action plan based on the user's emotional state and needs,

[0927] An output method for presenting the generated action plan to the user,

[0928] Support measures that provide guidance policies to help improve users' conversational skills,

[0929] Legal support measures that provide information related to legal proceedings,

[0930] A collaborative mechanism to collect and support participation in useful local activities based on the user's location,

[0931] Based on the analysis results, we provide a personalized life planning plan for each user, along with care support measures that improve convenience.

[0932] A system that includes this.

[0933] (Claim 2)

[0934] The system according to claim 1, comprising an analysis means that uses natural language processing technology to identify the emotional state of a user.

[0935] (Claim 3)

[0936] The system according to claim 1, further comprising means for updating the action plan presented to the user according to the current situation.

[0937] "Example 2 of combining an emotion engine"

[0938] (Claim 1)

[0939] A means for acquiring voice or text data from a user,

[0940] An analytical method that analyzes acquired data to identify the user's emotional state and needs,

[0941] A generation means for generating a customized action plan based on the user's emotional state and needs,

[0942] A means of presenting the generated action plan to the user,

[0943] A feedback analysis method for receiving user feedback and reflecting that information in the generation of the next plan,

[0944] A conversion means for converting input audio data into text data,

[0945] A system that includes this.

[0946] (Claim 2)

[0947] The system according to claim 1, further comprising an analysis means that uses a language analysis engine to identify the emotional state of a user.

[0948] (Claim 3)

[0949] The system according to claim 1, comprising an update means for updating the action plan presented to the user in real time and making adjustments based on the user's situation and feedback.

[0950] "Application example 2 when combining with an emotional engine"

[0951] (Claim 1)

[0952] An input means for acquiring voice or text data from the user,

[0953] An analytical tool that analyzes acquired data to identify the user's emotional state and needs, and provides support suggestions in natural language,

[0954] A generation means for generating a customized action plan based on the user's emotional state and needs,

[0955] An output method that presents the generated action plan to the user and provides support related to the living environment,

[0956] Support measures that provide guidance policies to help improve users' communication skills,

[0957] A means of coordination to facilitate collaboration with local organizations,

[0958] A system that includes this.

[0959] (Claim 2)

[0960] The system according to claim 1, comprising means for analyzing voice information when identifying the emotional state of a user.

[0961] (Claim 3)

[0962] The system according to claim 1, which updates the action plan presented to the user in real time and is used as a household assistant. [Explanation of Symbols]

[0963] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An input device that acquires voice or text data from the user, An analysis device that analyzes acquired data to identify the user's emotional state and needs, A generator that generates a customized action plan based on the user's emotional state and needs, An output device that presents the generated action plan to the user, A support device that provides communication guidelines to help improve users' conversational skills, A legal support device that provides information on legal procedures, A collaborative device that facilitates connection between users and related local communities, A system that includes this.

2. The system according to claim 1, further comprising an analysis means that uses a natural language processing engine to identify the emotional state of a user.

3. The system according to claim 1, further comprising an update means for updating the action plan presented to the user in real time.

Images