system

Abstract

We provide the system. [Solution] A means of converting and analyzing audio data, A means of evaluating and judging the living environment based on information collected from environmental sensors, A means of automatically sending notifications based on data, A means of generating and authenticating documents based on instructions, Means for electronically transmitting the aforementioned authenticated document, A means of extracting keywords from conversations and identifying related procedures using natural language processing technology, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In an aging society, many elderly people find administrative procedures conducted at local government offices cumbersome. In particular, for the elderly who live a passive life, they may not even be aware of the existence of such procedures. Furthermore, there is concern that information regarding procedure deadlines and methods may not reach the elderly appropriately, resulting in delays in necessary procedures and a decline in the quality of life. In such a situation, there is an increasing need for a system that allows the elderly to complete procedures naturally without making special efforts or being conscious.

Means for Solving the Problems

[0005] This invention provides a system that enables elderly people to naturally complete administrative procedures in their daily lives. Specifically, it utilizes a smart device that analyzes voice data to suggest administrative procedures, evaluates the living environment using data acquired from environmental sensors, and automatically sends reminders to the user at appropriate times. Furthermore, it provides more personalized alerts by sending reminders based on the user's health condition. In addition, the system reduces the burden on the elderly and supports appropriate procedures by automatically generating the necessary documents for procedures, accepting electronic signatures, and allowing document submission online.

[0006] "Voice data" refers to digital data collected from the voices of elderly individuals and stored or processed in an analyzable format.

[0007] A "sensor" is a device used to collect data about the environment, movements, and physical condition, and is used to understand the user's lifestyle patterns.

[0008] A "reminder" is a function that notifies users based on specific times or conditions, making them aware of procedures or things they need to check.

[0009] An "electronic signature" is a method of verifying the authenticity and issuer of a document by electronically signing it using digital signature technology.

[0010] "Online submission" refers to the process of submitting documents electronically via the internet, eliminating the need for physical submission in paper form. [Brief explanation of the drawing]

[0011] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3]This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0012] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0013] First, let's explain the terminology used in the following explanation.

[0014] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0015] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0016] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0017] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0018] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0019] [First Embodiment]

[0020] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0021] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0022] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0023] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0024] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0025] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0026] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0027] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0028] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0029] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0030] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0031] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0032] This invention provides technology for realizing a system that allows elderly people to efficiently complete administrative procedures without being consciously aware of them. The system operates around a server, terminals (smart devices and wearable devices), and the user.

[0033] First, the server collects voice data from the smart device and uses natural language processing technology to analyze keywords in the conversation. This allows it to identify the administrative procedures the user needs and suggest them at the appropriate time. For example, if the user says "pension procedures," the server uses this as a trigger to send information about those procedures to the device.

[0034] Regarding environmental sensor data, the device monitors the temperature, humidity, and operating conditions within the home and transmits this data to the server in real time. The server analyzes this data and learns the user's lifestyle patterns. This allows it to suggest actions at the most appropriate time. For example, it can send a reminder at the time the user usually wakes up, facilitating actions in a more natural way.

[0035] The server also sets automatic reminders for users based on procedure deadlines and required health data. The terminal notifies users of these reminders via smart devices or wearable devices. These reminders include details of the procedure and instructions on necessary actions.

[0036] Furthermore, the server automatically creates the necessary documents based on the user's past registration information. These documents are provided to the user via their device, and the user can electronically sign them on their smartphone or tablet. Once the electronic signature is complete, the server automatically sends the documents online to the appropriate recipient and confirms the completion of the procedure. This allows users to complete the process easily without physical travel or the hassle of filling out paperwork.

[0037] Thus, the present invention provides a system that supports administrative procedures in a way that seamlessly integrates into the daily lives of the elderly, simplifying the necessary steps to the greatest extent possible. In particular, it aims to reduce the burden on the elderly and provide a more secure living environment.

[0038] The following describes the processing flow.

[0039] Step 1:

[0040] The server analyzes audio data collected from everyday conversations to identify keywords related to administrative procedures. This analysis uses natural language processing technology to determine if an elderly person requires any kind of procedure.

[0041] Step 2:

[0042] The terminal (smart device) proposes identified procedures to elderly individuals based on instructions from the server. It then guides the user through a voice assistant, explaining the procedure outline and necessary actions.

[0043] Step 3:

[0044] The terminal (environmental sensor) records temperature, humidity, and activity data within the home and sends it to the server. Based on this data, the server learns the user's lifestyle patterns and determines the optimal timing for suggesting procedures.

[0045] Step 4:

[0046] The server identifies procedures with approaching deadlines based on past procedure history and health check data, and automatically generates reminders. These reminders include specific instructions regarding the procedures.

[0047] Step 5:

[0048] The device (smartphone or wearable device) sends reminders to the elderly. The user receives the notification and checks the necessary procedures on their tablet or smartphone.

[0049] Step 6:

[0050] The server automatically generates the necessary documents for the procedure and provides them to the user via the terminal. The user reviews the presented documents and electronically signs them on their smartphone or tablet screen.

[0051] Step 7:

[0052] The server, upon receiving the electronic signature, automatically sends the signed document to the designated online portal to confirm the completion of the procedure. This allows the user to complete the process without requiring any special action.

[0053] (Example 1)

[0054] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0055] The challenge is to alleviate the difficulties elderly people face in smoothly completing administrative procedures. Specifically, problems include the inability to grasp the necessary information for procedures, the difficulty in preparing and submitting documents in accordance with the schedule, and the difficulty in receiving timely notifications that take into account the health status of elderly people.

[0056] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0057] In this invention, the server includes means for analyzing voice information from a communication device and proposing relevant procedures, means for evaluating the living environment based on information acquired from multiple sensors and determining the appropriate timing for the procedures, and means for automatically sending notifications to the user based on the deadline for the procedures and health information. This makes it possible for elderly people to complete the procedures in a natural way without being conscious of it.

[0058] "Communication device" refers to a general device used to transmit voice information.

[0059] "Audio information" refers to the conversion of a user's voice into a format that can be processed as digital data.

[0060] "Suggestion" refers to the act of providing information about the procedures and methods that users should follow.

[0061] A "sensor" refers to a device used to monitor the environment or the health status of users.

[0062] "Information" refers to a broad dataset that includes data acquired from sensors.

[0063] "Living environment" refers to the physical or social environment in which users reside, and evaluating it includes data-driven understanding.

[0064] "Timing" refers to the optimal time when procedures or notifications should be carried out.

[0065] "Notification" refers to a message sent to inform a user of some kind of information.

[0066] "Documents" refer to written materials that organize or record the information necessary for a procedure.

[0067] "Electronic authentication" refers to the process of signing or verifying documents in a digital format.

[0068] A "communication network" refers to the entire infrastructure used for sending and receiving data.

[0069] This invention provides a concrete model for implementing a system that allows elderly people to efficiently complete administrative procedures without being consciously aware of them. The system primarily operates using a server and terminals (smart devices and wearable devices).

[0070] The server collects voice information from edge devices and converts it into digital data using a speech analysis API, which is speech recognition software. In this process, a natural language processing toolkit is used to extract relevant keywords from the transcribed voice data. For example, if a user says the words "pension procedures," the server uses that keyword as a trigger to proceed with the analysis and identify the necessary procedural information.

[0071] The terminal also monitors environmental information obtained from sensors, including temperature, humidity, and user activity. This data is transmitted to the server in real time, where machine learning algorithms are used to analyze the user's lifestyle patterns. Based on this analysis, the server notifies the user at the optimal time to ensure that they do not miss any opportunities for action.

[0072] Furthermore, the server automatically generates the necessary documents for the user's procedure and provides them to the user via the terminal. The user can electronically authenticate the documents on their smart device. After electronic authentication is complete, the server automatically submits the documents online to the recipient, completing the procedure. This entire process allows the user to complete the procedure easily without needing to travel or fill out paperwork.

[0073] For example, if an elderly person says something like, "What should I do about my pension procedures next month?" in a casual conversation, the server will pick up on the conversation, send relevant information to the terminal, and assist with the procedures. An example of a prompt sentence to input into the generating AI model is, "Please tell me how elderly people can complete their pension procedures in a natural way."

[0074] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0075] Step 1:

[0076] The server receives audio information transmitted from the terminal and converts it into text data using a speech analysis API. In this process, the server receives audio data as input and generates text information as output. This conversion process extracts specific keywords based on the words spoken by the user, preparing them for the next step.

[0077] Step 2:

[0078] The server analyzes text information using a natural language processing toolkit. This analysis infers the user's intent and identifies relevant administrative procedures and necessary information. The input is the text information generated in step 1, and the output is a list of identified administrative procedures. Specifically, if the server detects the keyword "pension procedures," it selects the corresponding information.

[0079] Step 3:

[0080] The device collects environmental data from multiple sensors within the home and transmits it to a server. It takes sensor information such as temperature, humidity, and activity as input, and sends this data to the server as output. This process establishes a foundation for the server to understand the user's lifestyle patterns and environmental conditions.

[0081] Step 4:

[0082] The server uses machine learning algorithms to analyze the user's daily rhythm based on the acquired environmental data. The input is the sensor information obtained in step 3, and the output is the result of a judgment regarding the appropriate timing. For example, the server identifies the time of day when the user usually starts their activities and plans notifications for procedures accordingly.

[0083] Step 5:

[0084] The server automatically generates notifications based on the user's procedure deadlines and health information, and sends them to the user via their device. Inputs include past registration information and analytical data, while output is a notification with a specific date and time. The notification includes procedure details and action instructions to help the user respond quickly.

[0085] Step 6:

[0086] The server automatically generates the necessary documents for the procedure and sends them to the terminal. The information to be entered is the procedure details identified in step 2, and the output is the automatically generated documents. The user reviews these documents on the terminal and performs electronic authentication.

[0087] Step 7:

[0088] The user performs electronic authentication on the documents provided on the terminal. The input is the document received in step 6, and the output is the authenticated document. Once authentication is complete, the server automatically sends the document online to the designated submission destination. The user's procedure is officially completed upon submission of the completed document.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] The aim is to enable elderly people to complete administrative procedures naturally while reducing their own burden. In particular, there is a need to propose procedures efficiently using everyday conversations and daily routines, and to enable the completion of procedures without the need for cumbersome paperwork or physical travel.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes means for converting and analyzing voice data, means for evaluating and judging the living environment based on information collected from environmental sensors, and means for automatically sending notifications based on the data. This allows the user to naturally receive information for performing necessary procedures through everyday conversation and to receive notifications prompting them to perform those procedures at the appropriate time.

[0094] "Means for converting and analyzing audio data" refers to a function that converts acoustic information into text information and analyzes its content to extract the user's intent and necessary procedures.

[0095] "A means of evaluating and judging the living environment based on information collected from environmental sensors" refers to a function that monitors data related to the living environment and uses that information to determine the optimal timing for the user's actions.

[0096] "Means of automatically sending notifications based on data" refers to a function that automatically sends procedures and reminders that the user needs to their device based on pre-set conditions.

[0097] "Means for generating and authenticating documents based on instructions" refers to a function that generates necessary documents based on user instructions and data, and then applies an electronic signature to them.

[0098] "Means of electronically transmitting authenticated documents" refers to the function that completes the process of transmitting signed documents to a designated recipient via the internet.

[0099] "A means of extracting keywords from conversations and identifying related procedures using natural language processing technology" refers to a technology that analyzes the content of a user's utterances and extracts keywords for identifying necessary procedures with high accuracy.

[0100] The system for implementing this invention combines speech recognition technology and natural language processing technology to streamline administrative procedures for the elderly. The system mainly consists of a server, smart devices and wearable devices as terminals, and the user.

[0101] The server converts speech data into text using a speech recognition library (e.g., Google® Speech-to-Text). This text data is then analyzed by a natural language processing framework (e.g., TENSORFLOW® or spaCy) to extract specific keywords from the user's speech. This allows the server to identify which administrative procedures are required. The server then notifies the user of relevant information and reminders about the procedures via their smart device.

[0102] On the device side, environmental sensors collect data on the living environment. This includes temperature, humidity, and the user's activity status. This sensor data is sent to a server using an IoT platform (e.g., AWS® IoT, Azure® IoT Hub) for analysis. Based on these analysis results, the server determines the optimal timing for the user to perform certain procedures. For example, it might send a reminder for administrative procedures when the user normally wakes up.

[0103] Users can use their smart devices to electronically sign documents and review necessary paperwork. The server automatically generates the required documents and sends them online to the recipient after signing. This allows elderly individuals to complete procedures easily and quickly without physical travel.

[0104] For example, if a user needs to submit a specific document each month, mentioning that document in a conversation will trigger a process where the server manages the submission deadline, sends reminders at the appropriate time, and guides the user through to the submission of the signed document.

[0105] An example of a prompt for a generative AI model is: "Develop a natural language processing model that extracts keywords related to administrative procedures from the everyday conversations of elderly people and provides reminders about those keywords at the optimal time." This can be used to improve system operation and develop further functions.

[0106] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0107] Step 1:

[0108] The server acquires audio data and converts it to text using a speech recognition library. The input is the user's voice data, and the output is the text version of this voice. In this conversion process, the sound wave data is analyzed and converted into the optimal word sequence by a language model.

[0109] Step 2:

[0110] The server applies natural language processing techniques to extract keywords from text data and identify the administrative procedures required by the user. The input is text data obtained through speech recognition, and the output is the identified keywords and related procedural information. Here, a text analysis algorithm evaluates language patterns and identifies highly relevant keywords.

[0111] Step 3:

[0112] The device's environmental sensors collect living environment data such as temperature and humidity and transmit it to the server. The input is physical sensor data, and the output is living environment information generated based on this data. The data from the sensors is converted into digital signals and uploaded to the server via the network.

[0113] Step 4:

[0114] The server determines the optimal timing for a procedure based on environmental data and the user's past behavioral patterns. The input is sensor data and past behavioral data, and the output is the determined optimal timing. A data analysis algorithm uses statistical methods to model the user's life patterns.

[0115] Step 5:

[0116] The server automatically generates a reminder and notifies the terminal based on the identified procedure and optimal timing information. The input is the identified procedure information and the optimal timing, while the output is a reminder message displayed on the user's terminal. The notification system delivers the reminder to the user through the terminal's display and notification sound.

[0117] Step 6:

[0118] The user reviews the necessary documents for the procedure on their smart device and provides an electronic signature. The input is document information provided by the server, and the output is the user's signed document. A signature field that can be operated with a finger is provided through the user interface.

[0119] Step 7:

[0120] The server sends the signed documents to the appropriate recipient via an online system. The input is the signed document, and the output is a submission completion notification. An electronic document management system is used to ensure secure data transfer to the relevant organizations.

[0121] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0122] This invention combines an emotional engine with a system that allows elderly people to naturally complete necessary administrative procedures in their daily lives, thereby providing users with more precise and appropriate procedural support. This system operates around a server, terminals (smart devices, wearable devices), and the user.

[0123] First, the server analyzes the voice data collected from the smart device and uses an emotion engine to infer the user's emotional state when identifying the relevant administrative procedures. If the user is stressed, the system adjusts the frequency and content of notifications; conversely, if the user is relaxed, it attempts to make proactive suggestions. In this way, the emotion engine captures the emotional nuances contained in the voice data and operates the system in a more user-friendly manner.

[0124] The device optimizes the timing of reminder notifications based on information obtained from daily conversations and lifestyle monitoring. Based on emotional monitoring, if it is determined that notifications may be stressful at a high frequency, it has the flexibility to change the notification method or cancel the notification altogether. Furthermore, by accumulating the user's long-term emotional tendencies, the emotional engine can develop a reminder strategy optimized for each individual user.

[0125] For example, when a device uses an emotion engine to make a suggestion such as "Pension procedures are approaching," if the server detects that the user is feeling anxious about the suggestion, it will provide additional support information and interaction with questions to alleviate that anxiety. On the other hand, if a positive response is detected, it will prompt the user to quickly move on to the next step in the procedure.

[0126] Furthermore, the server utilizes an emotion engine to track users' emotional states over extended periods, forming a feedback loop to optimize procedural approaches based on this historical data. This allows the system to deepen its understanding of individual users over time, enabling it to provide even greater convenience.

[0127] Thus, the primary objective of this invention is to provide seamless and flexible support for administrative procedures that elderly people must perform, thereby improving their sense of security in their daily lives.

[0128] The following describes the processing flow.

[0129] Step 1:

[0130] The server analyzes the user's everyday conversations through the voice assistant and identifies keywords related to administrative procedures. Simultaneously, it uses an emotion engine to analyze emotional cues contained in the voice.

[0131] Step 2:

[0132] The terminal notifies the user of the suggested procedures based on the analysis results from the server. Furthermore, it adjusts the tone and content of the notification to reflect the results of the emotion engine, providing the user with the most optimal information.

[0133] Step 3:

[0134] The server receives data from environmental sensors and wearable devices to evaluate the user's lifestyle patterns and health status. Based on the data from the emotion engine, it optimizes the timing and frequency of reminders to help users avoid stress.

[0135] Step 4:

[0136] Users check reminders on their devices and take action to proceed with the necessary procedures as needed. Sentimental data regarding approval or rejection of procedures is also collected and used to guide users through subsequent procedures.

[0137] Step 5:

[0138] The server automatically generates the necessary documents based on user instructions and supports document verification and electronic signing in a way that minimizes the burden as much as possible, following the inferences of the sentiment engine.

[0139] Step 6:

[0140] The terminal collects the signed document once the user has completed the electronic signature and sends it to the server. The server submits this document to the appropriate online service and allows the user to confirm that the process is complete.

[0141] Step 7:

[0142] The server leverages an emotion engine to track users' emotional tendencies over time, forming a feedback loop for improving future procedural suggestions and notification strategies. This makes the system even more effective and user-friendly.

[0143] (Example 2)

[0144] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0145] The complexity of administrative procedures faced by the elderly in their daily lives, coupled with the physical and mental burdens associated with aging, becomes a source of stress, leading to delays and incompleteness in the process. Therefore, meticulous support that takes their emotional state into consideration is necessary to ensure the seamless completion of these procedures. Notifications and reminders that take into account their health condition and living environment are also essential.

[0146] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0147] In this invention, the server includes means for analyzing the voice data of elderly people and providing administrative procedure options appropriate to the situation; means for determining the user's emotional state from the voice data using an emotion estimation function and adjusting the content and frequency of notifications; and means for accumulating emotional information and forming a feedback loop for formulating a procedural support plan tailored to individual users. This makes it possible for elderly people to complete administrative procedures efficiently and without burden.

[0148] "Audio data" refers to audio information collected from user speech, which is then converted into text through analysis.

[0149] "Administrative procedures" are the formal processes required to complete services and obligations provided by government agencies.

[0150] "Emotion estimation functionality" refers to a system's ability to analyze a user's voice data and other inputs to infer the emotional state the user is experiencing.

[0151] A "feedback loop" is an iterative process in which a system collects user responses and data, and uses that data to continuously improve the system's output and processes.

[0152] "Detection equipment" refers to devices used to acquire information, such as sensors and microphones.

[0153] "Information equipment" refers to devices designed for advanced communication and information processing, such as smartphones and computers.

[0154] A "wearable device" is a type of device that is worn on the body and is typically used to monitor the user's health status and behavior.

[0155] "Electronically approved" means officially authorized or accepted through digital means.

[0156] This invention is a system that combines voice analysis and emotion estimation to facilitate administrative procedures necessary for elderly people in their daily lives. The system consists of a server, terminals (smart devices, wearable devices), and the user.

[0157] The server collects audio data and converts it to text using natural language processing algorithms. General-purpose natural language processing libraries (e.g., NLTK and spaCy) are used for this purpose. The text converted from the audio data is then subjected to emotional analysis using a generative AI model. The generative AI model is input with prompts such as, "Analyze the user's feelings regarding pension procedures and suggest appropriate support methods."

[0158] The device detects content related to administrative procedures from everyday conversations and sends notifications to the user. A voice assistant (e.g., a general conversational agent) is used for these notifications. The device also has a function to dynamically change the timing and content of notifications based on the user's emotional state. This means that notifications are less frequent when the user is stressed, and more proactive when the user is relaxed.

[0159] For example, when a device uses sentiment analysis to send a reminder that "pension procedures are approaching," if the user expresses anxiety, the server prepares detailed guidance and answers to questions, providing support to reassure the user. Furthermore, the server accumulates sentiment data over a long period and, through feedback, provides an optimal procedural support process for each user. This allows elderly individuals to proceed with the procedures with peace of mind.

[0160] This system is expected to support administrative procedures in a format tailored to individual users, while reducing stress.

[0161] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0162] Step 1:

[0163] The device collects the user's everyday conversations as audio data. In this collection process, smart devices or wearable devices operate, recording audio data triggered by specific keywords or phrases. The input is the user's voice, which is stored on the device as a digital audio file. The output is the audio data used for subsequent processing.

[0164] Step 2:

[0165] The server receives audio data sent from the terminal and first converts it into text data using speech recognition software. Specifically, it uses an open-source natural language processing library to perform the audio-to-text conversion. The input is audio data, and the output is parseable text data. This step allows the extracted text to be used for the subsequent sentiment analysis.

[0166] Step 3:

[0167] The server inputs text data into a generating AI model and uses its sentiment estimation capabilities to analyze the user's emotional state. Specifically, it uses a sentiment analysis API to identify emotions such as positive, negative, and neutral from the text. The input is the transformed text data, and the output is metadata about the emotions (e.g., emotion type and intensity).

[0168] Step 4:

[0169] The server uses metadata related to emotions to determine the optimal action for administrative procedures based on the user's current emotional state. If the emotion is negative, it chooses an approach that mitigates the content of notifications and reminders. The input is data indicating the user's emotional state, and the output is an adjustment to the notification content or a suggestion for the procedure.

[0170] Step 5:

[0171] The device sends optimized notifications to the user based on instructions from the server. Specifically, it uses a voice assistant to send reminders such as, "A procedure is approaching. Are you ready?" The input is notification information optimized by the system, and the output is a voice or text notification to the user.

[0172] Step 6:

[0173] The server collects user feedback and stores sentiment data and usage history. This feedback is used to optimize future notifications and procedural suggestions. The input is user feedback, and the output is a dataset used for long-term improvement.

[0174] (Application Example 2)

[0175] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0176] The administrative procedures that elderly people face in their daily lives are complex, and the stress generated during this process becomes a mental and physical burden. Therefore, there is a need not only to simplify the procedures themselves, but also to provide support that takes into account the emotional state of the users. The present invention aims to provide an innovative method and system that enables elderly people to carry out administrative procedures with peace of mind.

[0177] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0178] In this invention, the server includes means for analyzing the user's speech information and suggesting relevant public procedures; means for evaluating the living space based on information obtained from ambient condition sensors and determining the appropriate timing for the procedures; and means for inferring the user's emotional state using an emotion analysis engine and adjusting the frequency and content of notifications. This makes it possible to provide procedural support at the optimal timing and in the most appropriate manner for the user.

[0179] "Speech information" refers to all information obtained through a user's voice and language, and includes specific instructions and intentions related to each individual user.

[0180] "Official procedures" refer to formal procedures based on rules established by the government or public institutions, and include necessary activities that ordinary citizens should undertake in accordance with laws and regulations.

[0181] An "ambient condition sensor" is a device used to measure physical environmental conditions, and is used to acquire and evaluate various data such as temperature, humidity, light intensity, and sound volume.

[0182] "Living space" refers to the place or environment where the user conducts their primary activities, and usually includes the user's home or residence, which is where they reside.

[0183] An "emotion analysis engine" refers to information processing technology and algorithms used to infer human emotions and psychological states from voice and language, and has the function of analyzing the emotional state of the user.

[0184] "Notifications" refer to messages or alerts intended to draw attention to or provide information to users, and are communicated through audio, text, or visual means.

[0185] "Emotional state" refers to the internal psychological and emotional state of the user, and includes changes in feelings such as stress, relief, joy, and anxiety.

[0186] This invention is a system designed to make administrative procedures and daily life management easier for the elderly. The server works in conjunction with smartphones and wearable devices and uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to analyze the user's speech information. Based on the analyzed data, it utilizes an emotion analysis engine (e.g., IBM Watson® Tone Analyzer) to infer the user's emotional state. The frequency and content of notifications are adjusted according to the emotional state, and reminders are sent at the appropriate time.

[0187] The terminal evaluates the living space through data from ambient sensor data and provides information about the user's living environment. This allows for the determination of the optimal timing for official procedures and enables personalized procedural support throughout the entire system.

[0188] For example, when sending a voice reminder saying, "Your pension application is due soon, are you making preparations?", if sentiment analysis detects that the user is showing anxiety, the system will ask, "Would you like to hear a more detailed explanation of the application procedure?" and provide supportive information to alleviate the user's anxiety. If a positive response is indicated, the system will provide specific guidance on how to proceed to the next step in the application process.

[0189] An example of a prompt message to a generating AI model is, "Please suggest countermeasures for when a user is feeling anxious." In response to this prompt, the system can generate appropriate countermeasures and implement them in user support.

[0190] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0191] Step 1:

[0192] The server receives audio information collected from smartphones and wearable devices. The input is audio data, which is converted into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). The output is the transcribed speech information. This process allows for the conversion of voice communication into text tailored to specific purposes.

[0193] Step 2:

[0194] The server inputs the transcribed speech information into an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to estimate the emotional state. The input is text data, and the output is an analysis result indicating the emotional state. Here, the user's psychological state is quantified by outputting various emotions as numerical information.

[0195] Step 3:

[0196] The device receives data acquired from ambient condition sensors and evaluates the living space. This input is environmental data, including information such as temperature and humidity. The output is the evaluation result regarding the living space. As part of the data calculation, it analyzes the impact of environmental conditions on the user's health and comfort.

[0197] Step 4:

[0198] The server determines the content and timing of notifications based on the emotion analysis results and the evaluation results of the living space. The input is the analyzed psychological state and environmental evaluation data, and the output is optimized reminder information. Specifically, it generates information and suggestions tailored to the user's mental and physical condition and sends them as reminders.

[0199] Step 5:

[0200] The user reviews the reminder received from the server and requests additional procedural information as needed. The input is the reminder information, and the output is the requested feedback. Specifically, the user can respond to the reminder using voice or touch gestures.

[0201] Step 6:

[0202] The server prepares prompt statements using a generative AI model to provide additional information and procedural support in response to user feedback. The input is the user's request, and the output is the generated prompt statement and the corresponding response information. This prompting allows the server to quickly and accurately provide the information the user needs.

[0203] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0204] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0205] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0206] [Second Embodiment]

[0207] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0208] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0209] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0210] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0211] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0212] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0213] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0214] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0215] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0216] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0217] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0218] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0219] This invention provides technology for realizing a system that allows elderly people to efficiently complete administrative procedures without being consciously aware of them. The system operates around a server, terminals (smart devices and wearable devices), and the user.

[0220] First, the server collects voice data from the smart device and uses natural language processing technology to analyze keywords in the conversation. This allows it to identify the administrative procedures the user needs and suggest them at the appropriate time. For example, if the user says "pension procedures," the server uses this as a trigger to send information about those procedures to the device.

[0221] Regarding environmental sensor data, the device monitors the temperature, humidity, and operating conditions within the home and transmits this data to the server in real time. The server analyzes this data and learns the user's lifestyle patterns. This allows it to suggest actions at the most appropriate time. For example, it can send a reminder at the time the user usually wakes up, facilitating actions in a more natural way.

[0222] The server also sets automatic reminders for users based on procedure deadlines and required health data. The terminal notifies users of these reminders via smart devices or wearable devices. These reminders include details of the procedure and instructions on necessary actions.

[0223] Furthermore, the server automatically creates the necessary documents based on the user's past registration information. These documents are provided to the user via their device, and the user can electronically sign them on their smartphone or tablet. Once the electronic signature is complete, the server automatically sends the documents online to the appropriate recipient and confirms the completion of the procedure. This allows users to complete the process easily without physical travel or the hassle of filling out paperwork.

[0224] Thus, the present invention provides a system that supports administrative procedures in a way that seamlessly integrates into the daily lives of the elderly, simplifying the necessary steps to the greatest extent possible. In particular, it aims to reduce the burden on the elderly and provide a more secure living environment.

[0225] The following describes the processing flow.

[0226] Step 1:

[0227] The server analyzes audio data collected from everyday conversations to identify keywords related to administrative procedures. This analysis uses natural language processing technology to determine if an elderly person requires any kind of procedure.

[0228] Step 2:

[0229] The terminal (smart device) proposes identified procedures to elderly individuals based on instructions from the server. It then guides the user through a voice assistant, explaining the procedure outline and necessary actions.

[0230] Step 3:

[0231] The terminal (environmental sensor) records temperature, humidity, and activity data within the home and sends it to the server. Based on this data, the server learns the user's lifestyle patterns and determines the optimal timing for suggesting procedures.

[0232] Step 4:

[0233] The server identifies procedures with approaching deadlines based on past procedure history and health check data, and automatically generates reminders. These reminders include specific instructions regarding the procedures.

[0234] Step 5:

[0235] The device (smartphone or wearable device) sends reminders to the elderly. The user receives the notification and checks the necessary procedures on their tablet or smartphone.

[0236] Step 6:

[0237] The server automatically generates the necessary documents for the procedure and provides them to the user via the terminal. The user reviews the presented documents and electronically signs them on their smartphone or tablet screen.

[0238] Step 7:

[0239] The server, upon receiving the electronic signature, automatically sends the signed document to the designated online portal to confirm the completion of the procedure. This allows the user to complete the process without requiring any special action.

[0240] (Example 1)

[0241] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0242] The challenge is to alleviate the difficulties elderly people face in smoothly completing administrative procedures. Specifically, problems include the inability to grasp the necessary information for procedures, the difficulty in preparing and submitting documents in accordance with the schedule, and the difficulty in receiving timely notifications that take into account the health status of elderly people.

[0243] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0244] In this invention, the server includes means for analyzing voice information from a communication device and proposing relevant procedures, means for evaluating the living environment based on information acquired from multiple sensors and determining the appropriate timing for the procedures, and means for automatically sending notifications to the user based on the deadline for the procedures and health information. This makes it possible for elderly people to complete the procedures in a natural way without being conscious of it.

[0245] "Communication device" refers to a general device used to transmit voice information.

[0246] "Audio information" refers to the conversion of a user's voice into a format that can be processed as digital data.

[0247] "Suggestion" refers to the act of providing information about the procedures and methods that users should follow.

[0248] A "sensor" refers to a device used to monitor the environment or the health status of users.

[0249] "Information" refers to a broad dataset that includes data acquired from sensors.

[0250] "Living environment" refers to the physical or social environment in which users reside, and evaluating it includes data-driven understanding.

[0251] "Timing" refers to the optimal time when procedures or notifications should be carried out.

[0252] "Notification" refers to a message sent to inform a user of some kind of information.

[0253] "Documents" refer to written materials that organize or record the information necessary for a procedure.

[0254] "Electronic authentication" refers to the process of signing or verifying documents in a digital format.

[0255] A "communication network" refers to the entire infrastructure used for sending and receiving data.

[0256] This invention provides a concrete model for implementing a system that allows elderly people to efficiently complete administrative procedures without being consciously aware of them. The system primarily operates using a server and terminals (smart devices and wearable devices).

[0257] The server collects voice information from edge devices and converts it into digital data using a speech analysis API, which is speech recognition software. In this process, a natural language processing toolkit is used to extract relevant keywords from the transcribed voice data. For example, if a user says the words "pension procedures," the server uses that keyword as a trigger to proceed with the analysis and identify the necessary procedural information.

[0258] The terminal also monitors environmental information obtained from sensors, including temperature, humidity, and user activity. This data is transmitted to the server in real time, where machine learning algorithms are used to analyze the user's lifestyle patterns. Based on this analysis, the server notifies the user at the optimal time to ensure that they do not miss any opportunities for action.

[0259] Furthermore, the server automatically generates the necessary documents for the user's procedure and provides them to the user via the terminal. The user can electronically authenticate the documents on their smart device. After electronic authentication is complete, the server automatically submits the documents online to the recipient, completing the procedure. This entire process allows the user to complete the procedure easily without needing to travel or fill out paperwork.

[0260] For example, if an elderly person says something like, "What should I do about my pension procedures next month?" in a casual conversation, the server will pick up on the conversation, send relevant information to the terminal, and assist with the procedures. An example of a prompt sentence to input into the generating AI model is, "Please tell me how elderly people can complete their pension procedures in a natural way."

[0261] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0262] Step 1:

[0263] The server receives audio information transmitted from the terminal and converts it into text data using a speech analysis API. In this process, the server receives audio data as input and generates text information as output. This conversion process extracts specific keywords based on the words spoken by the user, preparing them for the next step.

[0264] Step 2:

[0265] The server analyzes text information using a natural language processing toolkit. This analysis infers the user's intent and identifies relevant administrative procedures and necessary information. The input is the text information generated in step 1, and the output is a list of identified administrative procedures. Specifically, if the server detects the keyword "pension procedures," it selects the corresponding information.

[0266] Step 3:

[0267] The device collects environmental data from multiple sensors within the home and transmits it to a server. It takes sensor information such as temperature, humidity, and activity as input, and sends this data to the server as output. This process establishes a foundation for the server to understand the user's lifestyle patterns and environmental conditions.

[0268] Step 4:

[0269] The server uses machine learning algorithms to analyze the user's daily rhythm based on the acquired environmental data. The input is the sensor information obtained in step 3, and the output is the result of a judgment regarding the appropriate timing. For example, the server identifies the time of day when the user usually starts their activities and plans notifications for procedures accordingly.

[0270] Step 5:

[0271] The server automatically generates notifications based on the user's procedure deadlines and health information, and sends them to the user via their device. Inputs include past registration information and analytical data, while output is a notification with a specific date and time. The notification includes procedure details and action instructions to help the user respond quickly.

[0272] Step 6:

[0273] The server automatically generates the necessary documents for the procedure and sends them to the terminal. The information to be entered is the procedure details identified in step 2, and the output is the automatically generated documents. The user reviews these documents on the terminal and performs electronic authentication.

[0274] Step 7:

[0275] The user performs electronic authentication on the documents provided on the terminal. The input is the document received in step 6, and the output is the authenticated document. Once authentication is complete, the server automatically sends the document online to the designated submission destination. The user's procedure is officially completed upon submission of the completed document.

[0276] (Application Example 1)

[0277] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0278] The aim is to enable elderly people to complete administrative procedures naturally while reducing their own burden. In particular, there is a need to propose procedures efficiently using everyday conversations and daily routines, and to enable the completion of procedures without the need for cumbersome paperwork or physical travel.

[0279] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0280] In this invention, the server includes means for converting and analyzing voice data, means for evaluating and judging the living environment based on information collected from environmental sensors, and means for automatically sending notifications based on the data. This allows the user to naturally receive information for performing necessary procedures through everyday conversation and to receive notifications prompting them to perform those procedures at the appropriate time.

[0281] "Means for converting and analyzing audio data" refers to a function that converts acoustic information into text information and analyzes its content to extract the user's intent and necessary procedures.

[0282] "Means for evaluating and determining the living environment based on information collected from environmental sensors" refers to the function of monitoring data related to the living environment and determining the optimal timing of procedures for the user from that information.

[0283] "Means for automatically sending notifications based on data" refers to the function of automatically sending procedures and reminders required by the user to the terminal based on preset conditions.

[0284] "Means for generating a document based on an instruction and performing authentication" refers to the function of generating necessary documents based on the user's instructions and data and performing an electronic signature on them.

[0285] "Means for electronically sending an authenticated document" refers to the function of completing the process of sending a signed document to a predetermined submission destination via the Internet.

[0286] "Means for extracting keywords from a conversation and identifying related procedures using natural language processing technology" refers to the technology of analyzing the user's utterance content and accurately extracting keywords for identifying necessary procedures.

[0287] The system for implementing this invention is a system that combines voice recognition technology and natural language processing technology to streamline administrative procedures for the elderly. The system mainly consists of a server, smart devices and wearable devices as terminals, and users.

[0288] The server uses a voice recognition library (e.g., Google Speech-to-Text) to convert voice data into text. This text data is analyzed by a natural language processing framework (e.g., TensorFlow or spaCy) to extract specific keywords from the user's speech. Thereby, it is possible to identify which administrative procedures are required. Subsequently, the server notifies relevant information and procedure reminders via the user's smart device.

[0289] On the device side, environmental sensors collect data about the living environment. This includes temperature, humidity, and the user's activity status. This sensor data is sent to a server using an IoT platform (e.g., AWS IoT, Azure IoT Hub) for analysis. Based on these analysis results, the server determines the optimal timing for the user to perform certain procedures. For example, it might send a reminder for administrative procedures when the user normally wakes up.

[0290] Users can use their smart devices to electronically sign documents and review necessary paperwork. The server automatically generates the required documents and sends them online to the recipient after signing. This allows elderly individuals to complete procedures easily and quickly without physical travel.

[0291] For example, if a user needs to submit a specific document each month, mentioning that document in a conversation will trigger a process where the server manages the submission deadline, sends reminders at the appropriate time, and guides the user through to the submission of the signed document.

[0292] An example of a prompt for a generative AI model is: "Develop a natural language processing model that extracts keywords related to administrative procedures from the everyday conversations of elderly people and provides reminders about those keywords at the optimal time." This can be used to improve system operation and develop further functions.

[0293] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0294] Step 1:

[0295] The server acquires audio data and converts it to text using a speech recognition library. The input is the user's voice data, and the output is the text version of this voice. In this conversion process, the sound wave data is analyzed and converted into the optimal word sequence by a language model.

[0296] Step 2:

[0297] The server applies natural language processing techniques to extract keywords from text data and identify the administrative procedures required by the user. The input is text data obtained through speech recognition, and the output is the identified keywords and related procedural information. Here, a text analysis algorithm evaluates language patterns and identifies highly relevant keywords.

[0298] Step 3:

[0299] The device's environmental sensors collect living environment data such as temperature and humidity and transmit it to the server. The input is physical sensor data, and the output is living environment information generated based on this data. The data from the sensors is converted into digital signals and uploaded to the server via the network.

[0300] Step 4:

[0301] The server determines the optimal timing for a procedure based on environmental data and the user's past behavioral patterns. The input is sensor data and past behavioral data, and the output is the determined optimal timing. A data analysis algorithm uses statistical methods to model the user's life patterns.

[0302] Step 5:

[0303] The server automatically generates a reminder and notifies the terminal based on the identified procedure and optimal timing information. The input is the identified procedure information and the optimal timing, while the output is a reminder message displayed on the user's terminal. The notification system delivers the reminder to the user through the terminal's display and notification sound.

[0304] Step 6:

[0305] The user checks the documents required for the procedure on the smart device and performs an electronic signature. The input is the document information provided by the server, and the output is the user's signed document. A signature field operable by finger is provided through the user interface.

[0306] Step 7:

[0307] The server sends the signed document to the appropriate destination through the online system. The input is the signed document, and the output is a submission completion notice. An electronic document management system is used to transfer data to relevant institutions in a secure manner.

[0308] Furthermore, an emotion engine for estimating the user's emotions may be combined. That is, the specific processing unit 290 may estimate the user's emotions using the emotion identification model 59 and perform specific processing using the user's emotions.

[0309] The present invention provides more refined and appropriate procedure support to the user by combining an emotion engine with a system that allows the elderly to naturally complete the administrative procedures necessary in daily life. This system operates centering around the server, terminals (smart devices, wearable devices), and the user.

[0310] First, when the server analyzes the voice data collected from the smart device and identifies the related administrative procedure, it uses the emotion engine to infer the user's emotional state. When the user is feeling stressed, the system adjusts the frequency and content of the notifications. Conversely, when the user is relaxed, the system attempts positive suggestions. In this way, the emotion engine captures the emotional nuances contained in the voice data and operates the system in a more user-friendly manner.

[0311] The device optimizes the timing of reminder notifications based on information obtained from daily conversations and lifestyle monitoring. Based on emotional monitoring, if it is determined that notifications may be stressful at a high frequency, it has the flexibility to change the notification method or cancel the notification altogether. Furthermore, by accumulating the user's long-term emotional tendencies, the emotional engine can develop a reminder strategy optimized for each individual user.

[0312] For example, when a device uses an emotion engine to make a suggestion such as "Pension procedures are approaching," if the server detects that the user is feeling anxious about the suggestion, it will provide additional support information and interaction with questions to alleviate that anxiety. On the other hand, if a positive response is detected, it will prompt the user to quickly move on to the next step in the procedure.

[0313] Furthermore, the server utilizes an emotion engine to track users' emotional states over extended periods, forming a feedback loop to optimize procedural approaches based on this historical data. This allows the system to deepen its understanding of individual users over time, enabling it to provide even greater convenience.

[0314] Thus, the primary objective of this invention is to provide seamless and flexible support for administrative procedures that elderly people must perform, thereby improving their sense of security in their daily lives.

[0315] The following describes the processing flow.

[0316] Step 1:

[0317] The server analyzes the user's everyday conversations through the voice assistant and identifies keywords related to administrative procedures. Simultaneously, it uses an emotion engine to analyze emotional cues contained in the voice.

[0318] Step 2:

[0319] The terminal notifies the user of the suggested procedures based on the analysis results from the server. Furthermore, it adjusts the tone and content of the notification to reflect the results of the emotion engine, providing the user with the most optimal information.

[0320] Step 3:

[0321] The server receives data from environmental sensors and wearable devices to evaluate the user's lifestyle patterns and health status. Based on the data from the emotion engine, it optimizes the timing and frequency of reminders to help users avoid stress.

[0322] Step 4:

[0323] Users check reminders on their devices and take action to proceed with the necessary procedures as needed. Sentimental data regarding approval or rejection of procedures is also collected and used to guide users through subsequent procedures.

[0324] Step 5:

[0325] The server automatically generates the necessary documents based on user instructions and supports document verification and electronic signing in a way that minimizes the burden as much as possible, following the inferences of the sentiment engine.

[0326] Step 6:

[0327] The terminal collects the signed document once the user has completed the electronic signature and sends it to the server. The server submits this document to the appropriate online service and allows the user to confirm that the process is complete.

[0328] Step 7:

[0329] The server leverages an emotion engine to track users' emotional tendencies over time, forming a feedback loop for improving future procedural suggestions and notification strategies. This makes the system even more effective and user-friendly.

[0330] (Example 2)

[0331] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0332] The complexity of administrative procedures faced by the elderly in their daily lives, coupled with the physical and mental burdens associated with aging, becomes a source of stress, leading to delays and incompleteness in the process. Therefore, meticulous support that takes their emotional state into consideration is necessary to ensure the seamless completion of these procedures. Notifications and reminders that take into account their health condition and living environment are also essential.

[0333] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0334] In this invention, the server includes means for analyzing the voice data of elderly people and providing administrative procedure options appropriate to the situation; means for determining the user's emotional state from the voice data using an emotion estimation function and adjusting the content and frequency of notifications; and means for accumulating emotional information and forming a feedback loop for formulating a procedural support plan tailored to individual users. This makes it possible for elderly people to complete administrative procedures efficiently and without burden.

[0335] "Audio data" refers to audio information collected from user speech, which is then converted into text through analysis.

[0336] "Administrative procedures" are the formal processes required to complete services and obligations provided by government agencies.

[0337] "Emotion estimation functionality" refers to a system's ability to analyze a user's voice data and other inputs to infer the emotional state the user is experiencing.

[0338] A "feedback loop" is an iterative process in which a system collects user responses and data, and uses that data to continuously improve the system's output and processes.

[0339] "Detection equipment" refers to devices used to acquire information, such as sensors and microphones.

[0340] "Information equipment" refers to devices designed for advanced communication and information processing, such as smartphones and computers.

[0341] A "wearable device" is a type of device that is worn on the body and is typically used to monitor the user's health status and behavior.

[0342] "Electronically approved" means officially authorized or accepted through digital means.

[0343] This invention is a system that combines voice analysis and emotion estimation to facilitate administrative procedures necessary for elderly people in their daily lives. The system consists of a server, terminals (smart devices, wearable devices), and the user.

[0344] The server collects audio data and converts it to text using natural language processing algorithms. General-purpose natural language processing libraries (e.g., NLTK and spaCy) are used for this purpose. The text converted from the audio data is then subjected to emotional analysis using a generative AI model. The generative AI model is input with prompts such as, "Analyze the user's feelings regarding pension procedures and suggest appropriate support methods."

[0345] The device detects content related to administrative procedures from everyday conversations and sends notifications to the user. A voice assistant (e.g., a general conversational agent) is used for these notifications. The device also has a function to dynamically change the timing and content of notifications based on the user's emotional state. This means that notifications are less frequent when the user is stressed, and more proactive when the user is relaxed.

[0346] For example, when a device uses sentiment analysis to send a reminder that "pension procedures are approaching," if the user expresses anxiety, the server prepares detailed guidance and answers to questions, providing support to reassure the user. Furthermore, the server accumulates sentiment data over a long period and, through feedback, provides an optimal procedural support process for each user. This allows elderly individuals to proceed with the procedures with peace of mind.

[0347] This system is expected to support administrative procedures in a format tailored to individual users, while reducing stress.

[0348] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0349] Step 1:

[0350] The device collects the user's everyday conversations as audio data. In this collection process, smart devices or wearable devices operate, recording audio data triggered by specific keywords or phrases. The input is the user's voice, which is stored on the device as a digital audio file. The output is the audio data used for subsequent processing.

[0351] Step 2:

[0352] The server receives audio data sent from the terminal and first converts it into text data using speech recognition software. Specifically, it uses an open-source natural language processing library to perform the audio-to-text conversion. The input is audio data, and the output is parseable text data. This step allows the extracted text to be used for the subsequent sentiment analysis.

[0353] Step 3:

[0354] The server inputs text data into a generating AI model and uses its sentiment estimation capabilities to analyze the user's emotional state. Specifically, it uses a sentiment analysis API to identify emotions such as positive, negative, and neutral from the text. The input is the transformed text data, and the output is metadata about the emotions (e.g., emotion type and intensity).

[0355] Step 4:

[0356] The server uses metadata related to emotions to determine the optimal action for administrative procedures based on the user's current emotional state. If the emotion is negative, it chooses an approach that mitigates the content of notifications and reminders. The input is data indicating the user's emotional state, and the output is an adjustment to the notification content or a suggestion for the procedure.

[0357] Step 5:

[0358] The device sends optimized notifications to the user based on instructions from the server. Specifically, it uses a voice assistant to send reminders such as, "A procedure is approaching. Are you ready?" The input is notification information optimized by the system, and the output is a voice or text notification to the user.

[0359] Step 6:

[0360] The server collects user feedback and stores sentiment data and usage history. This feedback is used to optimize future notifications and procedural suggestions. The input is user feedback, and the output is a dataset used for long-term improvement.

[0361] (Application Example 2)

[0362] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0363] The administrative procedures that elderly people face in their daily lives are complex, and the stress generated during this process becomes a mental and physical burden. Therefore, there is a need not only to simplify the procedures themselves, but also to provide support that takes into account the emotional state of the users. The present invention aims to provide an innovative method and system that enables elderly people to carry out administrative procedures with peace of mind.

[0364] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0365] In this invention, the server includes means for analyzing the user's speech information and suggesting relevant public procedures; means for evaluating the living space based on information obtained from ambient condition sensors and determining the appropriate timing for the procedures; and means for inferring the user's emotional state using an emotion analysis engine and adjusting the frequency and content of notifications. This makes it possible to provide procedural support at the optimal timing and in the most appropriate manner for the user.

[0366] "Speech information" refers to all information obtained through a user's voice and language, and includes specific instructions and intentions related to each individual user.

[0367] "Official procedures" refer to formal procedures based on rules established by the government or public institutions, and include necessary activities that ordinary citizens should undertake in accordance with laws and regulations.

[0368] An "ambient condition sensor" is a device used to measure physical environmental conditions, and is used to acquire and evaluate various data such as temperature, humidity, light intensity, and sound volume.

[0369] "Living space" refers to the place or environment where the user conducts their primary activities, and usually includes the user's home or residence, which is where they reside.

[0370] An "emotion analysis engine" refers to information processing technology and algorithms used to infer human emotions and psychological states from voice and language, and has the function of analyzing the emotional state of the user.

[0371] "Notifications" refer to messages or alerts intended to draw attention to or provide information to users, and are communicated through audio, text, or visual means.

[0372] "Emotional state" refers to the internal psychological and emotional state of the user, and includes changes in feelings such as stress, relief, joy, and anxiety.

[0373] This invention is a system designed to make administrative procedures and daily life management easier for the elderly. The server works in conjunction with smartphones and wearable devices and uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to analyze the user's speech information. Based on the analyzed data, it utilizes an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to infer the user's emotional state. The frequency and content of notifications are adjusted according to the emotional state, and reminders are sent at the appropriate time.

[0374] The terminal evaluates the living space through data from ambient sensor data and provides information about the user's living environment. This allows for the determination of the optimal timing for official procedures and enables personalized procedural support throughout the entire system.

[0375] For example, when sending a voice reminder saying, "Your pension application is due soon, are you making preparations?", if sentiment analysis detects that the user is showing anxiety, the system will ask, "Would you like to hear a more detailed explanation of the application procedure?" and provide supportive information to alleviate the user's anxiety. If a positive response is indicated, the system will provide specific guidance on how to proceed to the next step in the application process.

[0376] An example of a prompt message to a generating AI model is, "Please suggest countermeasures for when a user is feeling anxious." In response to this prompt, the system can generate appropriate countermeasures and implement them in user support.

[0377] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0378] Step 1:

[0379] The server receives audio information collected from smartphones and wearable devices. The input is audio data, which is converted into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). The output is the transcribed speech information. This process allows for the conversion of voice communication into text tailored to specific purposes.

[0380] Step 2:

[0381] The server inputs the transcribed speech information into an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to estimate the emotional state. The input is text data, and the output is an analysis result indicating the emotional state. Here, the user's psychological state is quantified by outputting various emotions as numerical information.

[0382] Step 3:

[0383] The device receives data acquired from ambient condition sensors and evaluates the living space. This input is environmental data, including information such as temperature and humidity. The output is the evaluation result regarding the living space. As part of the data calculation, it analyzes the impact of environmental conditions on the user's health and comfort.

[0384] Step 4:

[0385] The server determines the content and timing of notifications based on the emotion analysis results and the evaluation results of the living space. The input is the analyzed psychological state and environmental evaluation data, and the output is optimized reminder information. Specifically, it generates information and suggestions tailored to the user's mental and physical condition and sends them as reminders.

[0386] Step 5:

[0387] The user reviews the reminder received from the server and requests additional procedural information as needed. The input is the reminder information, and the output is the requested feedback. Specifically, the user can respond to the reminder using voice or touch gestures.

[0388] Step 6:

[0389] The server prepares prompt statements using a generative AI model to provide additional information and procedural support in response to user feedback. The input is the user's request, and the output is the generated prompt statement and the corresponding response information. This prompting allows the server to quickly and accurately provide the information the user needs.

[0390] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0391] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0392] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0393] [Third Embodiment]

[0394] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0395] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0396] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0397] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0398] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0399] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0400] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0401] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0402] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0403] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0404] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0405] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0406] This invention provides technology for realizing a system that allows elderly people to efficiently complete administrative procedures without being consciously aware of them. The system operates around a server, terminals (smart devices and wearable devices), and the user.

[0407] First, the server collects voice data from the smart device and uses natural language processing technology to analyze keywords in the conversation. This allows it to identify the administrative procedures the user needs and suggest them at the appropriate time. For example, if the user says "pension procedures," the server uses this as a trigger to send information about those procedures to the device.

[0408] Regarding environmental sensor data, the device monitors the temperature, humidity, and operating conditions within the home and transmits this data to the server in real time. The server analyzes this data and learns the user's lifestyle patterns. This allows it to suggest actions at the most appropriate time. For example, it can send a reminder at the time the user usually wakes up, facilitating actions in a more natural way.

[0409] The server also sets automatic reminders for users based on procedure deadlines and required health data. The terminal notifies users of these reminders via smart devices or wearable devices. These reminders include details of the procedure and instructions on necessary actions.

[0410] Furthermore, the server automatically creates the necessary documents based on the user's past registration information. These documents are provided to the user via their device, and the user can electronically sign them on their smartphone or tablet. Once the electronic signature is complete, the server automatically sends the documents online to the appropriate recipient and confirms the completion of the procedure. This allows users to complete the process easily without physical travel or the hassle of filling out paperwork.

[0411] Thus, the present invention provides a system that supports administrative procedures in a way that seamlessly integrates into the daily lives of the elderly, simplifying the necessary steps to the greatest extent possible. In particular, it aims to reduce the burden on the elderly and provide a more secure living environment.

[0412] The following describes the processing flow.

[0413] Step 1:

[0414] The server analyzes audio data collected from everyday conversations to identify keywords related to administrative procedures. This analysis uses natural language processing technology to determine if an elderly person requires any kind of procedure.

[0415] Step 2:

[0416] The terminal (smart device) proposes identified procedures to elderly individuals based on instructions from the server. It then guides the user through a voice assistant, explaining the procedure outline and necessary actions.

[0417] Step 3:

[0418] The terminal (environmental sensor) records temperature, humidity, and activity data within the home and sends it to the server. Based on this data, the server learns the user's lifestyle patterns and determines the optimal timing for suggesting procedures.

[0419] Step 4:

[0420] The server identifies procedures with approaching deadlines based on past procedure history and health check data, and automatically generates reminders. These reminders include specific instructions regarding the procedures.

[0421] Step 5:

[0422] The device (smartphone or wearable device) sends reminders to the elderly. The user receives the notification and checks the necessary procedures on their tablet or smartphone.

[0423] Step 6:

[0424] The server automatically generates the necessary documents for the procedure and provides them to the user via the terminal. The user reviews the presented documents and electronically signs them on their smartphone or tablet screen.

[0425] Step 7:

[0426] The server, upon receiving the electronic signature, automatically sends the signed document to the designated online portal to confirm the completion of the procedure. This allows the user to complete the process without requiring any special action.

[0427] (Example 1)

[0428] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0429] The challenge is to alleviate the difficulties elderly people face in smoothly completing administrative procedures. Specifically, problems include the inability to grasp the necessary information for procedures, the difficulty in preparing and submitting documents in accordance with the schedule, and the difficulty in receiving timely notifications that take into account the health status of elderly people.

[0430] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0431] In this invention, the server includes means for analyzing voice information from a communication device and proposing relevant procedures, means for evaluating the living environment based on information acquired from multiple sensors and determining the appropriate timing for the procedures, and means for automatically sending notifications to the user based on the deadline for the procedures and health information. This makes it possible for elderly people to complete the procedures in a natural way without being conscious of it.

[0432] "Communication device" refers to a general device used to transmit voice information.

[0433] "Audio information" refers to the conversion of a user's voice into a format that can be processed as digital data.

[0434] "Suggestion" refers to the act of providing information about the procedures and methods that users should follow.

[0435] A "sensor" refers to a device used to monitor the environment or the health status of users.

[0436] "Information" refers to a broad dataset that includes data acquired from sensors.

[0437] "Living environment" refers to the physical or social environment in which users reside, and evaluating it includes data-driven understanding.

[0438] "Timing" refers to the optimal time when procedures or notifications should be carried out.

[0439] "Notification" refers to a message sent to inform a user of some kind of information.

[0440] "Documents" refer to written materials that organize or record the information necessary for a procedure.

[0441] "Electronic authentication" refers to the process of signing or verifying documents in a digital format.

[0442] A "communication network" refers to the entire infrastructure used for sending and receiving data.

[0443] This invention provides a concrete model for implementing a system that allows elderly people to efficiently complete administrative procedures without being consciously aware of them. The system primarily operates using a server and terminals (smart devices and wearable devices).

[0444] The server collects voice information from edge devices and converts it into digital data using a speech analysis API, which is speech recognition software. In this process, a natural language processing toolkit is used to extract relevant keywords from the transcribed voice data. For example, if a user says the words "pension procedures," the server uses that keyword as a trigger to proceed with the analysis and identify the necessary procedural information.

[0445] The terminal also monitors environmental information obtained from sensors, including temperature, humidity, and user activity. This data is transmitted to the server in real time, where machine learning algorithms are used to analyze the user's lifestyle patterns. Based on this analysis, the server notifies the user at the optimal time to ensure that they do not miss any opportunities for action.

[0446] Furthermore, the server automatically generates the necessary documents for the user's procedure and provides them to the user via the terminal. The user can electronically authenticate the documents on their smart device. After electronic authentication is complete, the server automatically submits the documents online to the recipient, completing the procedure. This entire process allows the user to complete the procedure easily without needing to travel or fill out paperwork.

[0447] For example, if an elderly person says something like, "What should I do about my pension procedures next month?" in a casual conversation, the server will pick up on the conversation, send relevant information to the terminal, and assist with the procedures. An example of a prompt sentence to input into the generating AI model is, "Please tell me how elderly people can complete their pension procedures in a natural way."

[0448] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0449] Step 1:

[0450] The server receives audio information transmitted from the terminal and converts it into text data using a speech analysis API. In this process, the server receives audio data as input and generates text information as output. This conversion process extracts specific keywords based on the words spoken by the user, preparing them for the next step.

[0451] Step 2:

[0452] The server analyzes text information using a natural language processing toolkit. This analysis infers the user's intent and identifies relevant administrative procedures and necessary information. The input is the text information generated in step 1, and the output is a list of identified administrative procedures. Specifically, if the server detects the keyword "pension procedures," it selects the corresponding information.

[0453] Step 3:

[0454] The device collects environmental data from multiple sensors within the home and transmits it to a server. It takes sensor information such as temperature, humidity, and activity as input, and sends this data to the server as output. This process establishes a foundation for the server to understand the user's lifestyle patterns and environmental conditions.

[0455] Step 4:

[0456] The server uses machine learning algorithms to analyze the user's daily rhythm based on the acquired environmental data. The input is the sensor information obtained in step 3, and the output is the result of a judgment regarding the appropriate timing. For example, the server identifies the time of day when the user usually starts their activities and plans notifications for procedures accordingly.

[0457] Step 5:

[0458] The server automatically generates notifications based on the user's procedure deadlines and health information, and sends them to the user via their device. Inputs include past registration information and analytical data, while output is a notification with a specific date and time. The notification includes procedure details and action instructions to help the user respond quickly.

[0459] Step 6:

[0460] The server automatically generates the necessary documents for the procedure and sends them to the terminal. The information to be entered is the procedure details identified in step 2, and the output is the automatically generated documents. The user reviews these documents on the terminal and performs electronic authentication.

[0461] Step 7:

[0462] The user performs electronic authentication on the documents provided on the terminal. The input is the document received in step 6, and the output is the authenticated document. Once authentication is complete, the server automatically sends the document online to the designated submission destination. The user's procedure is officially completed upon submission of the completed document.

[0463] (Application Example 1)

[0464] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0465] The aim is to enable elderly people to complete administrative procedures naturally while reducing their own burden. In particular, there is a need to propose procedures efficiently using everyday conversations and daily routines, and to enable the completion of procedures without the need for cumbersome paperwork or physical travel.

[0466] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0467] In this invention, the server includes means for converting and analyzing voice data, means for evaluating and judging the living environment based on information collected from environmental sensors, and means for automatically sending notifications based on the data. This allows the user to naturally receive information for performing necessary procedures through everyday conversation and to receive notifications prompting them to perform those procedures at the appropriate time.

[0468] "Means for converting and analyzing audio data" refers to a function that converts acoustic information into text information and analyzes its content to extract the user's intent and necessary procedures.

[0469] "A means of evaluating and judging the living environment based on information collected from environmental sensors" refers to a function that monitors data related to the living environment and uses that information to determine the optimal timing for the user's actions.

[0470] "Means of automatically sending notifications based on data" refers to a function that automatically sends procedures and reminders that the user needs to their device based on pre-set conditions.

[0471] "Means for generating and authenticating documents based on instructions" refers to a function that generates necessary documents based on user instructions and data, and then applies an electronic signature to them.

[0472] "Means of electronically transmitting authenticated documents" refers to the function that completes the process of transmitting signed documents to a designated recipient via the internet.

[0473] "A means of extracting keywords from conversations and identifying related procedures using natural language processing technology" refers to a technology that analyzes the content of a user's utterances and extracts keywords for identifying necessary procedures with high accuracy.

[0474] The system for implementing this invention combines speech recognition technology and natural language processing technology to streamline administrative procedures for the elderly. The system mainly consists of a server, smart devices and wearable devices as terminals, and the user.

[0475] The server converts speech data into text using a speech recognition library (e.g., Google Speech-to-Text). This text data is then analyzed by a natural language processing framework (e.g., TensorFlow or spaCy) to extract specific keywords from the user's speech. This allows the server to identify which administrative procedures are required. The server then notifies the user of relevant information and reminders about the procedures via their smart device.

[0476] On the device side, environmental sensors collect data about the living environment. This includes temperature, humidity, and the user's activity status. This sensor data is sent to a server using an IoT platform (e.g., AWS IoT, Azure IoT Hub) for analysis. Based on these analysis results, the server determines the optimal timing for the user to perform certain procedures. For example, it might send a reminder for administrative procedures when the user normally wakes up.

[0477] Users can use their smart devices to electronically sign documents and review necessary paperwork. The server automatically generates the required documents and sends them online to the recipient after signing. This allows elderly individuals to complete procedures easily and quickly without physical travel.

[0478] For example, if a user needs to submit a specific document each month, mentioning that document in a conversation will trigger a process where the server manages the submission deadline, sends reminders at the appropriate time, and guides the user through to the submission of the signed document.

[0479] An example of a prompt for a generative AI model is: "Develop a natural language processing model that extracts keywords related to administrative procedures from the everyday conversations of elderly people and provides reminders about those keywords at the optimal time." This can be used to improve system operation and develop further functions.

[0480] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0481] Step 1:

[0482] The server acquires audio data and converts it to text using a speech recognition library. The input is the user's voice data, and the output is the text version of this voice. In this conversion process, the sound wave data is analyzed and converted into the optimal word sequence by a language model.

[0483] Step 2:

[0484] The server applies natural language processing techniques to extract keywords from text data and identify the administrative procedures required by the user. The input is text data obtained through speech recognition, and the output is the identified keywords and related procedural information. Here, a text analysis algorithm evaluates language patterns and identifies highly relevant keywords.

[0485] Step 3:

[0486] The device's environmental sensors collect living environment data such as temperature and humidity and transmit it to the server. The input is physical sensor data, and the output is living environment information generated based on this data. The data from the sensors is converted into digital signals and uploaded to the server via the network.

[0487] Step 4:

[0488] The server determines the optimal timing for a procedure based on environmental data and the user's past behavioral patterns. The input is sensor data and past behavioral data, and the output is the determined optimal timing. A data analysis algorithm uses statistical methods to model the user's life patterns.

[0489] Step 5:

[0490] The server automatically generates a reminder and notifies the terminal based on the identified procedure and optimal timing information. The input is the identified procedure information and the optimal timing, while the output is a reminder message displayed on the user's terminal. The notification system delivers the reminder to the user through the terminal's display and notification sound.

[0491] Step 6:

[0492] The user reviews the necessary documents for the procedure on their smart device and provides an electronic signature. The input is document information provided by the server, and the output is the user's signed document. A signature field that can be operated with a finger is provided through the user interface.

[0493] Step 7:

[0494] The server sends the signed documents to the appropriate recipient via an online system. The input is the signed document, and the output is a submission completion notification. An electronic document management system is used to ensure secure data transfer to the relevant organizations.

[0495] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0496] This invention combines an emotional engine with a system that allows elderly people to naturally complete necessary administrative procedures in their daily lives, thereby providing users with more precise and appropriate procedural support. This system operates around a server, terminals (smart devices, wearable devices), and the user.

[0497] First, the server analyzes the voice data collected from the smart device and uses an emotion engine to infer the user's emotional state when identifying the relevant administrative procedures. If the user is stressed, the system adjusts the frequency and content of notifications; conversely, if the user is relaxed, it attempts to make proactive suggestions. In this way, the emotion engine captures the emotional nuances contained in the voice data and operates the system in a more user-friendly manner.

[0498] The device optimizes the timing of reminder notifications based on information obtained from daily conversations and lifestyle monitoring. Based on emotional monitoring, if it is determined that notifications may be stressful at a high frequency, it has the flexibility to change the notification method or cancel the notification altogether. Furthermore, by accumulating the user's long-term emotional tendencies, the emotional engine can develop a reminder strategy optimized for each individual user.

[0499] For example, when a device uses an emotion engine to make a suggestion such as "Pension procedures are approaching," if the server detects that the user is feeling anxious about the suggestion, it will provide additional support information and interaction with questions to alleviate that anxiety. On the other hand, if a positive response is detected, it will prompt the user to quickly move on to the next step in the procedure.

[0500] Furthermore, the server utilizes an emotion engine to track users' emotional states over extended periods, forming a feedback loop to optimize procedural approaches based on this historical data. This allows the system to deepen its understanding of individual users over time, enabling it to provide even greater convenience.

[0501] Thus, the primary objective of this invention is to provide seamless and flexible support for administrative procedures that elderly people must perform, thereby improving their sense of security in their daily lives.

[0502] The following describes the processing flow.

[0503] Step 1:

[0504] The server analyzes the user's everyday conversations through the voice assistant and identifies keywords related to administrative procedures. Simultaneously, it uses an emotion engine to analyze emotional cues contained in the voice.

[0505] Step 2:

[0506] The terminal notifies the user of the suggested procedures based on the analysis results from the server. Furthermore, it adjusts the tone and content of the notification to reflect the results of the emotion engine, providing the user with the most optimal information.

[0507] Step 3:

[0508] The server receives data from environmental sensors and wearable devices to evaluate the user's lifestyle patterns and health status. Based on the data from the emotion engine, it optimizes the timing and frequency of reminders to help users avoid stress.

[0509] Step 4:

[0510] Users check reminders on their devices and take action to proceed with the necessary procedures as needed. Sentimental data regarding approval or rejection of procedures is also collected and used to guide users through subsequent procedures.

[0511] Step 5:

[0512] The server automatically generates the necessary documents based on user instructions and supports document verification and electronic signing in a way that minimizes the burden as much as possible, following the inferences of the sentiment engine.

[0513] Step 6:

[0514] The terminal collects the signed document once the user has completed the electronic signature and sends it to the server. The server submits this document to the appropriate online service and allows the user to confirm that the process is complete.

[0515] Step 7:

[0516] The server leverages an emotion engine to track users' emotional tendencies over time, forming a feedback loop for improving future procedural suggestions and notification strategies. This makes the system even more effective and user-friendly.

[0517] (Example 2)

[0518] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0519] The complexity of administrative procedures faced by the elderly in their daily lives, coupled with the physical and mental burdens associated with aging, becomes a source of stress, leading to delays and incompleteness in the process. Therefore, meticulous support that takes their emotional state into consideration is necessary to ensure the seamless completion of these procedures. Notifications and reminders that take into account their health condition and living environment are also essential.

[0520] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0521] In this invention, the server includes means for analyzing the voice data of elderly people and providing administrative procedure options appropriate to the situation; means for determining the user's emotional state from the voice data using an emotion estimation function and adjusting the content and frequency of notifications; and means for accumulating emotional information and forming a feedback loop for formulating a procedural support plan tailored to individual users. This makes it possible for elderly people to complete administrative procedures efficiently and without burden.

[0522] "Audio data" refers to audio information collected from user speech, which is then converted into text through analysis.

[0523] "Administrative procedures" are the formal processes required to complete services and obligations provided by government agencies.

[0524] "Emotion estimation functionality" refers to a system's ability to analyze a user's voice data and other inputs to infer the emotional state the user is experiencing.

[0525] A "feedback loop" is an iterative process in which a system collects user responses and data, and uses that data to continuously improve the system's output and processes.

[0526] "Detection equipment" refers to devices used to acquire information, such as sensors and microphones.

[0527] "Information equipment" refers to devices designed for advanced communication and information processing, such as smartphones and computers.

[0528] A "wearable device" is a type of device that is worn on the body and is typically used to monitor the user's health status and behavior.

[0529] "Electronically approved" means officially authorized or accepted through digital means.

[0530] This invention is a system that combines voice analysis and emotion estimation to facilitate administrative procedures necessary for elderly people in their daily lives. The system consists of a server, terminals (smart devices, wearable devices), and the user.

[0531] The server collects audio data and converts it to text using natural language processing algorithms. General-purpose natural language processing libraries (e.g., NLTK and spaCy) are used for this purpose. The text converted from the audio data is then subjected to emotional analysis using a generative AI model. The generative AI model is input with prompts such as, "Analyze the user's feelings regarding pension procedures and suggest appropriate support methods."

[0532] The device detects content related to administrative procedures from everyday conversations and sends notifications to the user. A voice assistant (e.g., a general conversational agent) is used for these notifications. The device also has a function to dynamically change the timing and content of notifications based on the user's emotional state. This means that notifications are less frequent when the user is stressed, and more proactive when the user is relaxed.

[0533] For example, when a device uses sentiment analysis to send a reminder that "pension procedures are approaching," if the user expresses anxiety, the server prepares detailed guidance and answers to questions, providing support to reassure the user. Furthermore, the server accumulates sentiment data over a long period and, through feedback, provides an optimal procedural support process for each user. This allows elderly individuals to proceed with the procedures with peace of mind.

[0534] This system is expected to support administrative procedures in a format tailored to individual users, while reducing stress.

[0535] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0536] Step 1:

[0537] The device collects the user's everyday conversations as audio data. In this collection process, smart devices or wearable devices operate, recording audio data triggered by specific keywords or phrases. The input is the user's voice, which is stored on the device as a digital audio file. The output is the audio data used for subsequent processing.

[0538] Step 2:

[0539] The server receives audio data sent from the terminal and first converts it into text data using speech recognition software. Specifically, it uses an open-source natural language processing library to perform the audio-to-text conversion. The input is audio data, and the output is parseable text data. This step allows the extracted text to be used for the subsequent sentiment analysis.

[0540] Step 3:

[0541] The server inputs text data into a generating AI model and uses its sentiment estimation capabilities to analyze the user's emotional state. Specifically, it uses a sentiment analysis API to identify emotions such as positive, negative, and neutral from the text. The input is the transformed text data, and the output is metadata about the emotions (e.g., emotion type and intensity).

[0542] Step 4:

[0543] The server uses metadata related to emotions to determine the optimal action for administrative procedures based on the user's current emotional state. If the emotion is negative, it chooses an approach that mitigates the content of notifications and reminders. The input is data indicating the user's emotional state, and the output is an adjustment to the notification content or a suggestion for the procedure.

[0544] Step 5:

[0545] The device sends optimized notifications to the user based on instructions from the server. Specifically, it uses a voice assistant to send reminders such as, "A procedure is approaching. Are you ready?" The input is notification information optimized by the system, and the output is a voice or text notification to the user.

[0546] Step 6:

[0547] The server collects user feedback and stores sentiment data and usage history. This feedback is used to optimize future notifications and procedural suggestions. The input is user feedback, and the output is a dataset used for long-term improvement.

[0548] (Application Example 2)

[0549] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0550] The administrative procedures that elderly people face in their daily lives are complex, and the stress generated during this process becomes a mental and physical burden. Therefore, there is a need not only to simplify the procedures themselves, but also to provide support that takes into account the emotional state of the users. The present invention aims to provide an innovative method and system that enables elderly people to carry out administrative procedures with peace of mind.

[0551] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0552] In this invention, the server includes means for analyzing the user's speech information and suggesting relevant public procedures; means for evaluating the living space based on information obtained from ambient condition sensors and determining the appropriate timing for the procedures; and means for inferring the user's emotional state using an emotion analysis engine and adjusting the frequency and content of notifications. This makes it possible to provide procedural support at the optimal timing and in the most appropriate manner for the user.

[0553] "Speech information" refers to all information obtained through a user's voice and language, and includes specific instructions and intentions related to each individual user.

[0554] "Official procedures" refer to formal procedures based on rules established by the government or public institutions, and include necessary activities that ordinary citizens should undertake in accordance with laws and regulations.

[0555] An "ambient condition sensor" is a device used to measure physical environmental conditions, and is used to acquire and evaluate various data such as temperature, humidity, light intensity, and sound volume.

[0556] "Living space" refers to the place or environment where the user conducts their primary activities, and usually includes the user's home or residence, which is where they reside.

[0557] An "emotion analysis engine" refers to information processing technology and algorithms used to infer human emotions and psychological states from voice and language, and has the function of analyzing the emotional state of the user.

[0558] "Notifications" refer to messages or alerts intended to draw attention to or provide information to users, and are communicated through audio, text, or visual means.

[0559] "Emotional state" refers to the internal psychological and emotional state of the user, and includes changes in feelings such as stress, relief, joy, and anxiety.

[0560] This invention is a system designed to make administrative procedures and daily life management easier for the elderly. The server works in conjunction with smartphones and wearable devices and uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to analyze the user's speech information. Based on the analyzed data, it utilizes an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to infer the user's emotional state. The frequency and content of notifications are adjusted according to the emotional state, and reminders are sent at the appropriate time.

[0561] The terminal evaluates the living space through data from ambient sensor data and provides information about the user's living environment. This allows for the determination of the optimal timing for official procedures and enables personalized procedural support throughout the entire system.

[0562] For example, when sending a voice reminder saying, "Your pension application is due soon, are you making preparations?", if sentiment analysis detects that the user is showing anxiety, the system will ask, "Would you like to hear a more detailed explanation of the application procedure?" and provide supportive information to alleviate the user's anxiety. If a positive response is indicated, the system will provide specific guidance on how to proceed to the next step in the application process.

[0563] An example of a prompt message to a generating AI model is, "Please suggest countermeasures for when a user is feeling anxious." In response to this prompt, the system can generate appropriate countermeasures and implement them in user support.

[0564] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0565] Step 1:

[0566] The server receives audio information collected from smartphones and wearable devices. The input is audio data, which is converted into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). The output is the transcribed speech information. This process allows for the conversion of voice communication into text tailored to specific purposes.

[0567] Step 2:

[0568] The server inputs the transcribed speech information into an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to estimate the emotional state. The input is text data, and the output is an analysis result indicating the emotional state. Here, the user's psychological state is quantified by outputting various emotions as numerical information.

[0569] Step 3:

[0570] The device receives data acquired from ambient condition sensors and evaluates the living space. This input is environmental data, including information such as temperature and humidity. The output is the evaluation result regarding the living space. As part of the data calculation, it analyzes the impact of environmental conditions on the user's health and comfort.

[0571] Step 4:

[0572] The server determines the content and timing of notifications based on the emotion analysis results and the evaluation results of the living space. The input is the analyzed psychological state and environmental evaluation data, and the output is optimized reminder information. Specifically, it generates information and suggestions tailored to the user's mental and physical condition and sends them as reminders.

[0573] Step 5:

[0574] The user reviews the reminder received from the server and requests additional procedural information as needed. The input is the reminder information, and the output is the requested feedback. Specifically, the user can respond to the reminder using voice or touch gestures.

[0575] Step 6:

[0576] The server prepares prompt statements using a generative AI model to provide additional information and procedural support in response to user feedback. The input is the user's request, and the output is the generated prompt statement and the corresponding response information. This prompting allows the server to quickly and accurately provide the information the user needs.

[0577] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0578] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0579] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0580] [Fourth Embodiment]

[0581] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0582] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0583] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0584] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0585] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0586] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0587] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0588] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0589] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0590] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0591] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0592] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0593] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0594] This invention provides technology for realizing a system that allows elderly people to efficiently complete administrative procedures without being consciously aware of them. The system operates around a server, terminals (smart devices and wearable devices), and the user.

[0595] First, the server collects voice data from the smart device and uses natural language processing technology to analyze keywords in the conversation. This allows it to identify the administrative procedures the user needs and suggest them at the appropriate time. For example, if the user says "pension procedures," the server uses this as a trigger to send information about those procedures to the device.

[0596] Regarding environmental sensor data, the device monitors the temperature, humidity, and operating conditions within the home and transmits this data to the server in real time. The server analyzes this data and learns the user's lifestyle patterns. This allows it to suggest actions at the most appropriate time. For example, it can send a reminder at the time the user usually wakes up, facilitating actions in a more natural way.

[0597] The server also sets automatic reminders for users based on procedure deadlines and required health data. The terminal notifies users of these reminders via smart devices or wearable devices. These reminders include details of the procedure and instructions on necessary actions.

[0598] Furthermore, the server automatically creates the necessary documents based on the user's past registration information. These documents are provided to the user via their device, and the user can electronically sign them on their smartphone or tablet. Once the electronic signature is complete, the server automatically sends the documents online to the appropriate recipient and confirms the completion of the procedure. This allows users to complete the process easily without physical travel or the hassle of filling out paperwork.

[0599] Thus, the present invention provides a system that supports administrative procedures in a way that seamlessly integrates into the daily lives of the elderly, simplifying the necessary steps to the greatest extent possible. In particular, it aims to reduce the burden on the elderly and provide a more secure living environment.

[0600] The following describes the processing flow.

[0601] Step 1:

[0602] The server analyzes audio data collected from everyday conversations to identify keywords related to administrative procedures. This analysis uses natural language processing technology to determine if an elderly person requires any kind of procedure.

[0603] Step 2:

[0604] The terminal (smart device) proposes identified procedures to elderly individuals based on instructions from the server. It then guides the user through a voice assistant, explaining the procedure outline and necessary actions.

[0605] Step 3:

[0606] The terminal (environmental sensor) records temperature, humidity, and activity data within the home and sends it to the server. Based on this data, the server learns the user's lifestyle patterns and determines the optimal timing for suggesting procedures.

[0607] Step 4:

[0608] The server identifies procedures with approaching deadlines based on past procedure history and health check data, and automatically generates reminders. These reminders include specific instructions regarding the procedures.

[0609] Step 5:

[0610] The device (smartphone or wearable device) sends reminders to the elderly. The user receives the notification and checks the necessary procedures on their tablet or smartphone.

[0611] Step 6:

[0612] The server automatically generates the necessary documents for the procedure and provides them to the user via the terminal. The user reviews the presented documents and electronically signs them on their smartphone or tablet screen.

[0613] Step 7:

[0614] The server, upon receiving the electronic signature, automatically sends the signed document to the designated online portal to confirm the completion of the procedure. This allows the user to complete the process without requiring any special action.

[0615] (Example 1)

[0616] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0617] The challenge is to alleviate the difficulties elderly people face in smoothly completing administrative procedures. Specifically, problems include the inability to grasp the necessary information for procedures, the difficulty in preparing and submitting documents in accordance with the schedule, and the difficulty in receiving timely notifications that take into account the health status of elderly people.

[0618] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0619] In this invention, the server includes means for analyzing voice information from a communication device and proposing relevant procedures, means for evaluating the living environment based on information acquired from multiple sensors and determining the appropriate timing for the procedures, and means for automatically sending notifications to the user based on the deadline for the procedures and health information. This makes it possible for elderly people to complete the procedures in a natural way without being conscious of it.

[0620] "Communication device" refers to a general device used to transmit voice information.

[0621] "Audio information" refers to the conversion of a user's voice into a format that can be processed as digital data.

[0622] "Suggestion" refers to the act of providing information about the procedures and methods that users should follow.

[0623] A "sensor" refers to a device used to monitor the environment or the health status of users.

[0624] "Information" refers to a broad dataset that includes data acquired from sensors.

[0625] "Living environment" refers to the physical or social environment in which users reside, and evaluating it includes data-driven understanding.

[0626] "Timing" refers to the optimal time when procedures or notifications should be carried out.

[0627] "Notification" refers to a message sent to inform a user of some kind of information.

[0628] "Documents" refer to written materials that organize or record the information necessary for a procedure.

[0629] "Electronic authentication" refers to the process of signing or verifying documents in a digital format.

[0630] A "communication network" refers to the entire infrastructure used for sending and receiving data.

[0631] This invention provides a concrete model for implementing a system that allows elderly people to efficiently complete administrative procedures without being consciously aware of them. The system primarily operates using a server and terminals (smart devices and wearable devices).

[0632] The server collects voice information from edge devices and converts it into digital data using a speech analysis API, which is speech recognition software. In this process, a natural language processing toolkit is used to extract relevant keywords from the transcribed voice data. For example, if a user says the words "pension procedures," the server uses that keyword as a trigger to proceed with the analysis and identify the necessary procedural information.

[0633] The terminal also monitors environmental information obtained from sensors, including temperature, humidity, and user activity. This data is transmitted to the server in real time, where machine learning algorithms are used to analyze the user's lifestyle patterns. Based on this analysis, the server notifies the user at the optimal time to ensure that they do not miss any opportunities for action.

[0634] Furthermore, the server automatically generates the necessary documents for the user's procedure and provides them to the user via the terminal. The user can electronically authenticate the documents on their smart device. After electronic authentication is complete, the server automatically submits the documents online to the recipient, completing the procedure. This entire process allows the user to complete the procedure easily without needing to travel or fill out paperwork.

[0635] For example, if an elderly person says something like, "What should I do about my pension procedures next month?" in a casual conversation, the server will pick up on the conversation, send relevant information to the terminal, and assist with the procedures. An example of a prompt sentence to input into the generating AI model is, "Please tell me how elderly people can complete their pension procedures in a natural way."

[0636] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0637] Step 1:

[0638] The server receives audio information transmitted from the terminal and converts it into text data using a speech analysis API. In this process, the server receives audio data as input and generates text information as output. This conversion process extracts specific keywords based on the words spoken by the user, preparing them for the next step.

[0639] Step 2:

[0640] The server analyzes text information using a natural language processing toolkit. This analysis infers the user's intent and identifies relevant administrative procedures and necessary information. The input is the text information generated in step 1, and the output is a list of identified administrative procedures. Specifically, if the server detects the keyword "pension procedures," it selects the corresponding information.

[0641] Step 3:

[0642] The device collects environmental data from multiple sensors within the home and transmits it to a server. It takes sensor information such as temperature, humidity, and activity as input, and sends this data to the server as output. This process establishes a foundation for the server to understand the user's lifestyle patterns and environmental conditions.

[0643] Step 4:

[0644] The server uses machine learning algorithms to analyze the user's daily rhythm based on the acquired environmental data. The input is the sensor information obtained in step 3, and the output is the result of a judgment regarding the appropriate timing. For example, the server identifies the time of day when the user usually starts their activities and plans notifications for procedures accordingly.

[0645] Step 5:

[0646] The server automatically generates notifications based on the user's procedure deadlines and health information, and sends them to the user via their device. Inputs include past registration information and analytical data, while output is a notification with a specific date and time. The notification includes procedure details and action instructions to help the user respond quickly.

[0647] Step 6:

[0648] The server automatically generates the necessary documents for the procedure and sends them to the terminal. The information to be entered is the procedure details identified in step 2, and the output is the automatically generated documents. The user reviews these documents on the terminal and performs electronic authentication.

[0649] Step 7:

[0650] The user performs electronic authentication on the documents provided on the terminal. The input is the document received in step 6, and the output is the authenticated document. Once authentication is complete, the server automatically sends the document online to the designated submission destination. The user's procedure is officially completed upon submission of the completed document.

[0651] (Application Example 1)

[0652] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0653] The aim is to enable elderly people to complete administrative procedures naturally while reducing their own burden. In particular, there is a need to propose procedures efficiently using everyday conversations and daily routines, and to enable the completion of procedures without the need for cumbersome paperwork or physical travel.

[0654] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0655] In this invention, the server includes means for converting and analyzing voice data, means for evaluating and judging the living environment based on information collected from environmental sensors, and means for automatically sending notifications based on the data. This allows the user to naturally receive information for performing necessary procedures through everyday conversation and to receive notifications prompting them to perform those procedures at the appropriate time.

[0656] "Means for converting and analyzing audio data" refers to a function that converts acoustic information into text information and analyzes its content to extract the user's intent and necessary procedures.

[0657] "A means of evaluating and judging the living environment based on information collected from environmental sensors" refers to a function that monitors data related to the living environment and uses that information to determine the optimal timing for the user's actions.

[0658] "Means of automatically sending notifications based on data" refers to a function that automatically sends procedures and reminders that the user needs to their device based on pre-set conditions.

[0659] "Means for generating and authenticating documents based on instructions" refers to a function that generates necessary documents based on user instructions and data, and then applies an electronic signature to them.

[0660] "Means of electronically transmitting authenticated documents" refers to the function that completes the process of transmitting signed documents to a designated recipient via the internet.

[0661] "A means of extracting keywords from conversations and identifying related procedures using natural language processing technology" refers to a technology that analyzes the content of a user's utterances and extracts keywords for identifying necessary procedures with high accuracy.

[0662] The system for implementing this invention combines speech recognition technology and natural language processing technology to streamline administrative procedures for the elderly. The system mainly consists of a server, smart devices and wearable devices as terminals, and the user.

[0663] The server converts speech data into text using a speech recognition library (e.g., Google Speech-to-Text). This text data is then analyzed by a natural language processing framework (e.g., TensorFlow or spaCy) to extract specific keywords from the user's speech. This allows the server to identify which administrative procedures are required. The server then notifies the user of relevant information and reminders about the procedures via their smart device.

[0664] On the device side, environmental sensors collect data about the living environment. This includes temperature, humidity, and the user's activity status. This sensor data is sent to a server using an IoT platform (e.g., AWS IoT, Azure IoT Hub) for analysis. Based on these analysis results, the server determines the optimal timing for the user to perform certain procedures. For example, it might send a reminder for administrative procedures when the user normally wakes up.

[0665] Users can use their smart devices to electronically sign documents and review necessary paperwork. The server automatically generates the required documents and sends them online to the recipient after signing. This allows elderly individuals to complete procedures easily and quickly without physical travel.

[0666] For example, if a user needs to submit a specific document each month, mentioning that document in a conversation will trigger a process where the server manages the submission deadline, sends reminders at the appropriate time, and guides the user through to the submission of the signed document.

[0667] An example of a prompt for a generative AI model is: "Develop a natural language processing model that extracts keywords related to administrative procedures from the everyday conversations of elderly people and provides reminders about those keywords at the optimal time." This can be used to improve system operation and develop further functions.

[0668] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0669] Step 1:

[0670] The server acquires audio data and converts it to text using a speech recognition library. The input is the user's voice data, and the output is the text version of this voice. In this conversion process, the sound wave data is analyzed and converted into the optimal word sequence by a language model.

[0671] Step 2:

[0672] The server applies natural language processing techniques to extract keywords from text data and identify the administrative procedures required by the user. The input is text data obtained through speech recognition, and the output is the identified keywords and related procedural information. Here, a text analysis algorithm evaluates language patterns and identifies highly relevant keywords.

[0673] Step 3:

[0674] The device's environmental sensors collect living environment data such as temperature and humidity and transmit it to the server. The input is physical sensor data, and the output is living environment information generated based on this data. The data from the sensors is converted into digital signals and uploaded to the server via the network.

[0675] Step 4:

[0676] The server determines the optimal timing for a procedure based on environmental data and the user's past behavioral patterns. The input is sensor data and past behavioral data, and the output is the determined optimal timing. A data analysis algorithm uses statistical methods to model the user's life patterns.

[0677] Step 5:

[0678] The server automatically generates a reminder and notifies the terminal based on the identified procedure and optimal timing information. The input is the identified procedure information and the optimal timing, while the output is a reminder message displayed on the user's terminal. The notification system delivers the reminder to the user through the terminal's display and notification sound.

[0679] Step 6:

[0680] The user reviews the necessary documents for the procedure on their smart device and provides an electronic signature. The input is document information provided by the server, and the output is the user's signed document. A signature field that can be operated with a finger is provided through the user interface.

[0681] Step 7:

[0682] The server sends the signed documents to the appropriate recipient via an online system. The input is the signed document, and the output is a submission completion notification. An electronic document management system is used to ensure secure data transfer to the relevant organizations.

[0683] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0684] This invention combines an emotional engine with a system that allows elderly people to naturally complete necessary administrative procedures in their daily lives, thereby providing users with more precise and appropriate procedural support. This system operates around a server, terminals (smart devices, wearable devices), and the user.

[0685] First, the server analyzes the voice data collected from the smart device and uses an emotion engine to infer the user's emotional state when identifying the relevant administrative procedures. If the user is stressed, the system adjusts the frequency and content of notifications; conversely, if the user is relaxed, it attempts to make proactive suggestions. In this way, the emotion engine captures the emotional nuances contained in the voice data and operates the system in a more user-friendly manner.

[0686] The device optimizes the timing of reminder notifications based on information obtained from daily conversations and lifestyle monitoring. Based on emotional monitoring, if it is determined that notifications may be stressful at a high frequency, it has the flexibility to change the notification method or cancel the notification altogether. Furthermore, by accumulating the user's long-term emotional tendencies, the emotional engine can develop a reminder strategy optimized for each individual user.

[0687] For example, when a device uses an emotion engine to make a suggestion such as "Pension procedures are approaching," if the server detects that the user is feeling anxious about the suggestion, it will provide additional support information and interaction with questions to alleviate that anxiety. On the other hand, if a positive response is detected, it will prompt the user to quickly move on to the next step in the procedure.

[0688] Furthermore, the server utilizes an emotion engine to track users' emotional states over extended periods, forming a feedback loop to optimize procedural approaches based on this historical data. This allows the system to deepen its understanding of individual users over time, enabling it to provide even greater convenience.

[0689] Thus, the primary objective of this invention is to provide seamless and flexible support for administrative procedures that elderly people must perform, thereby improving their sense of security in their daily lives.

[0690] The following describes the processing flow.

[0691] Step 1:

[0692] The server analyzes the user's everyday conversations through the voice assistant and identifies keywords related to administrative procedures. Simultaneously, it uses an emotion engine to analyze emotional cues contained in the voice.

[0693] Step 2:

[0694] The terminal notifies the user of the suggested procedures based on the analysis results from the server. Furthermore, it adjusts the tone and content of the notification to reflect the results of the emotion engine, providing the user with the most optimal information.

[0695] Step 3:

[0696] The server receives data from environmental sensors and wearable devices to evaluate the user's lifestyle patterns and health status. Based on the data from the emotion engine, it optimizes the timing and frequency of reminders to help users avoid stress.

[0697] Step 4:

[0698] Users check reminders on their devices and take action to proceed with the necessary procedures as needed. Sentimental data regarding approval or rejection of procedures is also collected and used to guide users through subsequent procedures.

[0699] Step 5:

[0700] The server automatically generates the necessary documents based on user instructions and supports document verification and electronic signing in a way that minimizes the burden as much as possible, following the inferences of the sentiment engine.

[0701] Step 6:

[0702] The terminal collects the signed document once the user has completed the electronic signature and sends it to the server. The server submits this document to the appropriate online service and allows the user to confirm that the process is complete.

[0703] Step 7:

[0704] The server leverages an emotion engine to track users' emotional tendencies over time, forming a feedback loop for improving future procedural suggestions and notification strategies. This makes the system even more effective and user-friendly.

[0705] (Example 2)

[0706] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0707] The complexity of administrative procedures faced by the elderly in their daily lives, coupled with the physical and mental burdens associated with aging, becomes a source of stress, leading to delays and incompleteness in the process. Therefore, meticulous support that takes their emotional state into consideration is necessary to ensure the seamless completion of these procedures. Notifications and reminders that take into account their health condition and living environment are also essential.

[0708] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0709] In this invention, the server includes means for analyzing the voice data of elderly people and providing administrative procedure options appropriate to the situation; means for determining the user's emotional state from the voice data using an emotion estimation function and adjusting the content and frequency of notifications; and means for accumulating emotional information and forming a feedback loop for formulating a procedural support plan tailored to individual users. This makes it possible for elderly people to complete administrative procedures efficiently and without burden.

[0710] "Audio data" refers to audio information collected from user speech, which is then converted into text through analysis.

[0711] "Administrative procedures" are the formal processes required to complete services and obligations provided by government agencies.

[0712] "Emotion estimation functionality" refers to a system's ability to analyze a user's voice data and other inputs to infer the emotional state the user is experiencing.

[0713] A "feedback loop" is an iterative process in which a system collects user responses and data, and uses that data to continuously improve the system's output and processes.

[0714] "Detection equipment" refers to devices used to acquire information, such as sensors and microphones.

[0715] "Information equipment" refers to devices designed for advanced communication and information processing, such as smartphones and computers.

[0716] A "wearable device" is a type of device that is worn on the body and is typically used to monitor the user's health status and behavior.

[0717] "Electronically approved" means officially authorized or accepted through digital means.

[0718] This invention is a system that combines voice analysis and emotion estimation to facilitate administrative procedures necessary for elderly people in their daily lives. The system consists of a server, terminals (smart devices, wearable devices), and the user.

[0719] The server collects audio data and converts it to text using natural language processing algorithms. General-purpose natural language processing libraries (e.g., NLTK and spaCy) are used for this purpose. The text converted from the audio data is then subjected to emotional analysis using a generative AI model. The generative AI model is input with prompts such as, "Analyze the user's feelings regarding pension procedures and suggest appropriate support methods."

[0720] The device detects content related to administrative procedures from everyday conversations and sends notifications to the user. A voice assistant (e.g., a general conversational agent) is used for these notifications. The device also has a function to dynamically change the timing and content of notifications based on the user's emotional state. This means that notifications are less frequent when the user is stressed, and more proactive when the user is relaxed.

[0721] For example, when a device uses sentiment analysis to send a reminder that "pension procedures are approaching," if the user expresses anxiety, the server prepares detailed guidance and answers to questions, providing support to reassure the user. Furthermore, the server accumulates sentiment data over a long period and, through feedback, provides an optimal procedural support process for each user. This allows elderly individuals to proceed with the procedures with peace of mind.

[0722] This system is expected to support administrative procedures in a format tailored to individual users, while reducing stress.

[0723] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0724] Step 1:

[0725] The device collects the user's everyday conversations as audio data. In this collection process, smart devices or wearable devices operate, recording audio data triggered by specific keywords or phrases. The input is the user's voice, which is stored on the device as a digital audio file. The output is the audio data used for subsequent processing.

[0726] Step 2:

[0727] The server receives audio data sent from the terminal and first converts it into text data using speech recognition software. Specifically, it uses an open-source natural language processing library to perform the audio-to-text conversion. The input is audio data, and the output is parseable text data. This step allows the extracted text to be used for the subsequent sentiment analysis.

[0728] Step 3:

[0729] The server inputs text data into a generating AI model and uses its sentiment estimation capabilities to analyze the user's emotional state. Specifically, it uses a sentiment analysis API to identify emotions such as positive, negative, and neutral from the text. The input is the transformed text data, and the output is metadata about the emotions (e.g., emotion type and intensity).

[0730] Step 4:

[0731] The server uses metadata related to emotions to determine the optimal action for administrative procedures based on the user's current emotional state. If the emotion is negative, it chooses an approach that mitigates the content of notifications and reminders. The input is data indicating the user's emotional state, and the output is an adjustment to the notification content or a suggestion for the procedure.

[0732] Step 5:

[0733] The device sends optimized notifications to the user based on instructions from the server. Specifically, it uses a voice assistant to send reminders such as, "A procedure is approaching. Are you ready?" The input is notification information optimized by the system, and the output is a voice or text notification to the user.

[0734] Step 6:

[0735] The server collects user feedback and stores sentiment data and usage history. This feedback is used to optimize future notifications and procedural suggestions. The input is user feedback, and the output is a dataset used for long-term improvement.

[0736] (Application Example 2)

[0737] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0738] The administrative procedures that elderly people face in their daily lives are complex, and the stress generated during this process becomes a mental and physical burden. Therefore, there is a need not only to simplify the procedures themselves, but also to provide support that takes into account the emotional state of the users. The present invention aims to provide an innovative method and system that enables elderly people to carry out administrative procedures with peace of mind.

[0739] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0740] In this invention, the server includes means for analyzing the user's speech information and suggesting relevant public procedures; means for evaluating the living space based on information obtained from ambient condition sensors and determining the appropriate timing for the procedures; and means for inferring the user's emotional state using an emotion analysis engine and adjusting the frequency and content of notifications. This makes it possible to provide procedural support at the optimal timing and in the most appropriate manner for the user.

[0741] "Speech information" refers to all information obtained through a user's voice and language, and includes specific instructions and intentions related to each individual user.

[0742] "Official procedures" refer to formal procedures based on rules established by the government or public institutions, and include necessary activities that ordinary citizens should undertake in accordance with laws and regulations.

[0743] An "ambient condition sensor" is a device used to measure physical environmental conditions, and is used to acquire and evaluate various data such as temperature, humidity, light intensity, and sound volume.

[0744] "Living space" refers to the place or environment where the user conducts their primary activities, and usually includes the user's home or residence, which is where they reside.

[0745] An "emotion analysis engine" refers to information processing technology and algorithms used to infer human emotions and psychological states from voice and language, and has the function of analyzing the emotional state of the user.

[0746] "Notifications" refer to messages or alerts intended to draw attention to or provide information to users, and are communicated through audio, text, or visual means.

[0747] "Emotional state" refers to the internal psychological and emotional state of the user, and includes changes in feelings such as stress, relief, joy, and anxiety.

[0748] This invention is a system designed to make administrative procedures and daily life management easier for the elderly. The server works in conjunction with smartphones and wearable devices and uses a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to analyze the user's speech information. Based on the analyzed data, it utilizes an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to infer the user's emotional state. The frequency and content of notifications are adjusted according to the emotional state, and reminders are sent at the appropriate time.

[0749] The terminal evaluates the living space through data from ambient sensor data and provides information about the user's living environment. This allows for the determination of the optimal timing for official procedures and enables personalized procedural support throughout the entire system.

[0750] For example, when sending a voice reminder saying, "Your pension application is due soon, are you making preparations?", if sentiment analysis detects that the user is showing anxiety, the system will ask, "Would you like to hear a more detailed explanation of the application procedure?" and provide supportive information to alleviate the user's anxiety. If a positive response is indicated, the system will provide specific guidance on how to proceed to the next step in the application process.

[0751] An example of a prompt message to a generating AI model is, "Please suggest countermeasures for when a user is feeling anxious." In response to this prompt, the system can generate appropriate countermeasures and implement them in user support.

[0752] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0753] Step 1:

[0754] The server receives audio information collected from smartphones and wearable devices. The input is audio data, which is converted into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text API). The output is the transcribed speech information. This process allows for the conversion of voice communication into text tailored to specific purposes.

[0755] Step 2:

[0756] The server inputs the transcribed speech information into an emotion analysis engine (e.g., IBM Watson Tone Analyzer) to estimate the emotional state. The input is text data, and the output is an analysis result indicating the emotional state. Here, the user's psychological state is quantified by outputting various emotions as numerical information.

[0757] Step 3:

[0758] The device receives data acquired from ambient condition sensors and evaluates the living space. This input is environmental data, including information such as temperature and humidity. The output is the evaluation result regarding the living space. As part of the data calculation, it analyzes the impact of environmental conditions on the user's health and comfort.

[0759] Step 4:

[0760] The server determines the content and timing of notifications based on the emotion analysis results and the evaluation results of the living space. The input is the analyzed psychological state and environmental evaluation data, and the output is optimized reminder information. Specifically, it generates information and suggestions tailored to the user's mental and physical condition and sends them as reminders.

[0761] Step 5:

[0762] The user reviews the reminder received from the server and requests additional procedural information as needed. The input is the reminder information, and the output is the requested feedback. Specifically, the user can respond to the reminder using voice or touch gestures.

[0763] Step 6:

[0764] The server prepares prompt statements using a generative AI model to provide additional information and procedural support in response to user feedback. The input is the user's request, and the output is the generated prompt statement and the corresponding response information. This prompting allows the server to quickly and accurately provide the information the user needs.

[0765] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0766] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0767] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0768] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0769] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0770] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0771] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0772] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0773] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0774] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0775] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0776] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0777] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0778] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0779] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0780] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0781] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0782] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0783] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0784] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0785] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0786] The following is further disclosed regarding the embodiments described above.

[0787] (Claim 1)

[0788] A processing device that analyzes voice data of elderly people and proposes relevant administrative procedures,

[0789] A processing means that evaluates the living environment based on data acquired from multiple environmental sensors and determines the appropriate timing for the procedure,

[0790] A processing method that automatically sends reminders to users based on procedure deadlines and health status,

[0791] Based on the instructions from the user, a processing means automatically generates the necessary documents for the procedure and accepts electronic signatures,

[0792] A processing means for submitting the aforementioned electronically signed document online,

[0793] A system that includes this.

[0794] (Claim 2)

[0795] The system according to claim 1, wherein the smart device has a voice assistant function and means for processing the daily conversations of elderly people.

[0796] (Claim 3)

[0797] The system according to claim 1, wherein the wearable device has means for periodically measuring the user's health status and reflecting this in the transmission of the reminder.

[0798] "Example 1"

[0799] (Claim 1)

[0800] A means of analyzing voice information from a communication device and proposing related procedures,

[0801] A means of evaluating the living environment based on information obtained from multiple sensors and determining the appropriate timing for procedures,

[0802] A means of automatically sending notifications to users based on procedure deadlines and health information,

[0803] Based on the instructions from the aforementioned user, a means for automatically generating the necessary documents for the procedure and accepting electronic authentication,

[0804] A means of submitting the aforementioned electronically authenticated document via a communication network,

[0805] A system that includes this.

[0806] (Claim 2)

[0807] The system according to claim 1, wherein the communication device has a voice function and means for processing the user's everyday conversation.

[0808] (Claim 3)

[0809] The system according to claim 1, wherein the wearable device has means for periodically measuring the user's health and reflecting this in the notification transmission.

[0810] "Application Example 1"

[0811] (Claim 1)

[0812] A means of converting and analyzing audio data,

[0813] A means of evaluating and judging the living environment based on information collected from environmental sensors,

[0814] A means of automatically sending notifications based on data,

[0815] A means of generating and authenticating documents based on instructions,

[0816] Means for electronically transmitting the aforementioned authenticated document,

[0817] A means of extracting keywords from conversations and identifying related procedures using natural language processing technology,

[0818] A system that includes this.

[0819] (Claim 2)

[0820] The system according to claim 1, which has a voice interface function and means for processing everyday language data.

[0821] (Claim 3)

[0822] The system according to claim 1, wherein a portable device has means for periodically acquiring the user's health information and associating it with the transmission of the notification.

[0823] "Example 2 of combining an emotion engine"

[0824] (Claim 1)

[0825] A processing device that analyzes voice data of elderly people and provides administrative procedure options appropriate to the situation,

[0826] A processing means that uses an emotion estimation function to determine the user's emotional state from voice data and adjust the content and frequency of notifications,

[0827] A processing means that accumulates emotional information and forms a feedback loop for formulating procedural support plans tailored to individual users,

[0828] A processing means that evaluates the living environment based on data collected from multiple detection devices and determines the optimal timing for procedural notification,

[0829] A processing system that automatically creates necessary documents based on user instructions and accepts electronic approval,

[0830] A processing means for transmitting the aforementioned approved documents via the Internet,

[0831] A system that includes this.

[0832] (Claim 2)

[0833] The system according to claim 1, wherein the information device has a dialogue function and means for processing the daily conversations of elderly people.

[0834] (Claim 3)

[0835] The system according to claim 1, wherein the wearable device has means for periodically acquiring the user's health information and reflecting it in the notification means.

[0836] "Application example 2 when combining with an emotional engine"

[0837] (Claim 1)

[0838] A processing device that analyzes speech information from elderly people and proposes relevant public procedures,

[0839] A processing means that evaluates the living space based on information acquired from multiple ambient condition sensors and determines the appropriate timing for the procedure,

[0840] A processing mechanism that automatically sends notifications to users based on the deadline for procedures and their health status,

[0841] A processing means that automatically generates the necessary documents for the procedure based on the instructions from the user and accepts electronic signatures,

[0842] A processing means for submitting the electronically signed document online,

[0843] A processing means that uses an emotion analysis engine to estimate the user's emotional state and adjusts the frequency and content of notifications,

[0844] A system that includes this.

[0845] (Claim 2)

[0846] The system according to claim 1, wherein the electronic device has a voice support function and means for processing the daily conversations of an elderly person.

[0847] (Claim 3)

[0848] The system according to claim 1, wherein the wearable device has means for periodically measuring the user's health status and reflecting this in the notification transmission. [Explanation of symbols]

[0849] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of converting and analyzing audio data, A means of evaluating and judging the living environment based on information collected from environmental sensors, A means of automatically sending notifications based on data, A means of generating and authenticating documents based on instructions, Means for electronically transmitting the aforementioned authenticated document, A means of extracting keywords from conversations and identifying related procedures using natural language processing technology, A system that includes this.

2. The system according to claim 1, which has a voice interface function and means for processing everyday language data.

3. The system according to claim 1, wherein a portable device has means for periodically acquiring the user's health information and associating it with the transmission of the notification.

Images