System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving input information of a user; means for analyzing the input information and generating an action plan; means for scheduling a notification based on the action plan; and means for transmitting the notification to a terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] People with developmental disabilities often face unique challenges in their daily lives. Specifically, they may have difficulty managing time, completing tasks, and communicating with others. This can have a negative impact on their social and workplace performance. The objective of the present invention is to support the characteristics and needs of people with developmental disabilities and provide an environment in which they can live their daily lives more smoothly. [Means for solving the problem]

[0005] To solve this problem, the present invention provides the following means: a system including means for accepting user input information, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, and means for sending notifications to a terminal. The system also includes means for setting and saving user profile information and providing support tailored to the user's characteristics, means for converting voice input into text data and extracting appropriate actions, means for generating and notifying reminders based on actions, means for generating and providing communication support messages based on the user's characteristics, and means for providing notifications via a smart device. This system enables people with developmental disabilities to lead more independent daily and social lives.

[0006] "User" refers to any individual who uses this system.

[0007] "Input Information" refers to the voice or text data provided by the user to the system.

[0008] "Analysis" refers to the process of understanding input information and extracting the necessary information.

[0009] An "action plan" refers to a schedule or task list generated based on user input.

[0010] "Notification" refers to a message sent by the system to the user informing them of the timing of the execution of the action plan.

[0011] "Device" means a device used by the system to send or display notifications, including smartphones, smart glasses, wireless earphones, etc.

[0012] "Profile Information" refers to data about a user's personal information, characteristics, and needs.

[0013] "Preservation" refers to the process of storing data and making it available for retrieval when needed.

[0014] "Traits" refers to the individual characteristics and needs associated with developmental disorders such as ADHD and ASD.

[0015] "Support" refers to the help and services provided to users to help them function better in their daily lives.

[0016] "Voice input" refers to a method in which a user provides data by speaking to a system.

[0017] "Text data" refers to data that has been converted from voice input into text information.

[0018] "Action" refers to the specific actions taken by the user regarding their schedule or task.

[0019] A "reminder" is a notification that reminds the user at a pre-set time.

[0020] "Communication support message" refers to a message generated by the system to facilitate communication between users and others.

[0021] "Smart Device" means a device that can connect to the Internet and receive and display notifications from the system. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] MODE FOR CARRYING OUT THE INVENTION

[0044] System Overview

[0045] This invention is a support system for solving problems faced by users with developmental disabilities in their daily and social lives. This system accepts and analyzes user input, generates an action plan, and presents notifications at appropriate times, providing an environment in which users can live more smoothly.

[0046] Handling of user profiles

[0047] server

[0048] When a user first launches an application, the server provides a screen for setting user profile information.

[0049] Users enter their personal information (e.g., name, age, characteristics of developmental disabilities, etc.), needs related to those characteristics, and support methods.

[0050] Terminal

[0051] The device stores the entered profile information locally and sends it to the server.

[0052] The server stores the received information in a database and generates a profile based on individual characteristics.

[0053] Generate an action plan

[0054] User

[0055] The user inputs daily schedules and tasks. This can be done by voice input or text input. For example, "I have a meeting at 1pm."

[0056] Terminal

[0057] The device converts voice input into text data and analyzes it to extract key information, which identifies specific times and action items.

[0058] server

[0059] It receives the analyzed text data and generates an action plan (pre-meeting preparation, reminders, related tasks, etc.) based on the user's characteristics and needs.

[0060] The action plan will include specific meeting times, materials to be presented, and necessary preparations.

[0061] Terminal

[0062] The generated action plan is sent to the device and stored locally, where the user can review it and adjust it if necessary.

[0063] Notifications and Reminders

[0064] server

[0065] Based on the generated action plan, schedule the timing of the notification (e.g., 5 minutes before the meeting).

[0066] The notification information is sent to the terminal.

[0067] Terminal

[0068] The reminder will be triggered at the scheduled time, and a notification will be displayed to the user via their device (smartphone, smart glasses, wireless earphones, etc.).

[0069] Communication Support

[0070] User

[0071] When a user needs assistance communicating with others, they can input a specific example (e.g., a conversation with their boss) using voice or text input.

[0072] Terminal

[0073] The device converts the voice into text data and analyzes it.

[0074] server

[0075] Based on the analyzed text data, the system generates an optimal communication support message for the user, such as "Hello, boss. I have a request for you today."

[0076] Terminal

[0077] The generated message is sent to the terminal and provided to the user, who can refer to it for smooth communication.

[0078] Specific examples

[0079] Example 1: Generating an action plan

[0080] A user speaks, "I have a meeting at 1pm."

[0081] The device converts the speech into text and sends it to the server.

[0082] The server analyzes the received text data and generates an action plan (such as sending a reminder five minutes before a meeting or a notification to prepare materials).

[0083] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[0084] Example 2: Communication support

[0085] A user enters the text "I have trouble talking to my boss."

[0086] The terminal analyzes the received text and sends it to the server.

[0087] The server generates a communication support message ("Hello, boss. I have a request for you today.") based on the analyzed data.

[0088] The terminal displays the generated message to the user to support communication.

[0089] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics of the user.

[0090] The processing flow will be explained below.

[0091] User profile settings

[0092] Step 1:

[0093] User The user launches the application for the first time and accesses the profile information input screen.

[0094] Step 2:

[0095] User Enter name, age, characteristics of developmental disorder (e.g., ADHD, ASD, LD, etc.), and support needs related to the characteristics.

[0096] Step 3:

[0097] Terminal The entered information is temporarily saved in the terminal's local storage.

[0098] Step 4:

[0099] The information stored in the device's local storage is sent to the server.

[0100] Step 5:

[0101] Server: Stores the received profile information in a database and generates a profile based on the characteristics.

[0102] Generate an action plan

[0103] Step 1:

[0104] User: Enter daily schedules and tasks (e.g., I have a meeting at 1:00 p.m.) using voice or text.

[0105] Step 2:

[0106] Device Converts voice input into text data and analyzes it to extract key information (e.g., date and time, action items).

[0107] Step 3:

[0108] The terminal sends the analyzed text data to the server.

[0109] Step 4:

[0110] Server: Based on the received text data, it generates an action plan tailored to each individual's characteristics and needs, including a timeline, preparation list, reminders, etc.

[0111] Step 5:

[0112] Server: Sends the generated action plan to the terminal.

[0113] Step 6:

[0114] The device stores the received action plan in local storage, notifies the user, and provides the user with the option to adjust the plan if necessary.

[0115] Notifications and Reminders

[0116] Step 1:

[0117] Server Based on the action plan, schedule the notification timing for each action (e.g., a reminder 5 minutes before a meeting).

[0118] Step 2:

[0119] Server: Sends scheduled notification information to the terminal.

[0120] Step 3:

[0121] Device Check the action plan stored in local storage based on the notification timing.

[0122] Step 4:

[0123] Device: A reminder is generated based on the notification timing, and a notification is displayed or audible to the user via a smartphone, smart glasses, or wireless earphones (for example, at 12:55, a notification is sent saying, "It's five minutes before the meeting. Please prepare your materials.").

[0124] Communication Support

[0125] Step 1:

[0126] User When there is something that requires support in communicating with others (e.g., a conversation with a boss), the user inputs it by voice or text.

[0127] Step 2:

[0128] The device converts voice input into text data, analyzes it, and extracts specific needs.

[0129] Step 3:

[0130] The terminal transmits the extracted needs to the server.

[0131] Step 4:

[0132] Server: Based on the received data, it generates the most appropriate communication support message for the user (e.g., "Hello, boss. I have a request for you today.").

[0133] Step 5:

[0134] The server sends the generated support message to the terminal.

[0135] Step 6:

[0136] Terminal: Displays received messages to the user, helping the user to communicate smoothly based on the messages.

[0137] Example 1

[0138] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0139] Users with developmental disabilities face challenges in their daily and social lives, including schedule management and smooth communication. These challenges can cause difficulties in daily and social activities if users do not receive appropriate support. The present invention aims to provide an effective solution to these challenges.

[0140] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0141] In this invention, the server includes means for accepting user input information, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, means for locally saving the generated action plan, and means for displaying notifications to the user. This enables an action plan to be automatically generated based on information input by the user and notifications to be sent at appropriate times. The server also includes means for setting and saving user profile information and means for providing support tailored to the user's characteristics. This enables detailed support tailored to the characteristics of each individual user. The server also includes means for converting voice input into text data, means for analyzing the text data and extracting appropriate actions, and means for generating a communication support message optimized for the user. This also enables smooth communication support through voice input.

[0142] "User" refers to an individual with a developmental disability who uses the system.

[0143] "Input information" refers to data that a user provides to the system, and includes, for example, schedules, tasks, profile information, and the like.

[0144] "Analysis" refers to the processing of data based on input information to understand the content and generate relevant action plans and notifications.

[0145] An "action plan" is a list of schedules and tasks generated based on input information, and is intended to support users' daily and social lives.

[0146] "Notification" means information provided to a user at an appropriate time based on an action plan, including reminders and alerts.

[0147] "Schedule" refers to the time and order in which notifications are set based on an action plan.

[0148] A "terminal" is a hardware device for running the system, including smartphones and tablets.

[0149] "Local storage" refers to the temporary or long-term storage of data within a device.

[0150] "Profile information" refers to personal information provided by the user to the system, including name, age, and characteristics of the developmental disorder.

[0151] "Support" refers to providing assistance tailored to the characteristics and needs of users.

[0152] "Voice input" refers to the means by which a user provides information to a system using speech.

[0153] "Text data" refers to data obtained by converting voice input into text information.

[0154] "Communication support messages" are messages generated based on analyzed data to help users communicate smoothly with others.

[0155] This invention is a system for solving problems faced by users with developmental disabilities in their daily and social lives. This system accepts and analyzes information input by the user, generates an action plan, and presents notifications at appropriate times, thereby providing an environment in which the user can live more smoothly.

[0156] Configuration and Operation Procedures

[0157] 1. Handling of User Profiles

[0158] When a user first launches the application, the server provides a user profile information setup screen, where the user enters their name, age, characteristics of their developmental disorder, support needs, etc. The device then stores the entered profile information locally and sends it to the server. The server then stores the received information in a database and generates a profile based on each individual's characteristics.

[0159] 2. Generate an action plan

[0160] The user inputs daily schedules and tasks using voice or text. For example, "I have a meeting at 1 p.m." The device converts the voice input into text data and analyzes it to extract key information. This analysis identifies specific times and action items. The server receives the analyzed text data and generates an action plan (pre-meeting preparations, reminders, and related tasks) based on the user's characteristics and needs. The generated action plan may include, for example, "send a reminder 5 minutes before the meeting and a notification to prepare materials." The action plan is then sent to the device and stored locally. The user can review it and adjust it as needed.

[0161] 3. Notifications and Reminders

[0162] The server schedules the notification based on the generated action plan. For example, it sets a reminder five minutes before a meeting. The notification information is sent to the device. The device triggers the reminder at the scheduled time, and a notification is displayed to the user via their smartphone or other device.

[0163] 4. Communication support

[0164] When a user needs support in communicating with others, they input a specific example (e.g., the contents of a conversation with their boss). This is done by voice input or text input. The device converts the voice into text data and analyzes it. The server generates an optimal communication support message for the user based on the analyzed text data. For example, it generates a specific message such as, "Hello, boss. I have a request for you today." The device displays the generated message to the user to support communication.

[0165] Specific examples

[0166] Example 1: Generating an action plan

[0167] A user speaks, "I have a meeting at 1pm."

[0168] The device converts the speech into text and sends it to the server.

[0169] The server analyzes the received text data and generates an action plan (such as sending a reminder five minutes before a meeting or a notification to prepare materials).

[0170] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[0171] Example 2: Communication support

[0172] A user enters the text "I have trouble talking to my boss."

[0173] The terminal analyzes the received text and sends it to the server.

[0174] The server generates a communication support message ("Hello, boss. I have a request for you today.") based on the analyzed data.

[0175] The terminal displays the generated message to the user to support communication.

[0176] Prompt Sentence Examples

[0177] "Please explain in natural language the process of dictating that there will be a meeting at 1pm."

[0178] "Please explain in detail the process for generating support messages for users who are uncomfortable speaking with their managers."

[0179] Through these steps, the system of the present invention provides an environment in which users with developmental disabilities can overcome challenges in their daily and social lives and live more smoothly.

[0180] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0181] Step 1:

[0182] The user initially launches the application, which starts the system operation and provides an input that displays a profile setting screen for the user.

[0183] Step 2:

[0184] The server provides a user profile information setup screen, allowing the user to input their name, age, characteristics of their developmental disorder, support needs, etc. This input leads to an output that captures the user's basic personal information.

[0185] Step 3:

[0186] The user enters personal information such as name, age, and characteristics of the developmental disorder. This input provides the system with basic data to provide support tailored to the user's characteristics. This data is then used in later steps to generate an action plan and support messages tailored to the user's characteristics.

[0187] Step 4:

[0188] The device locally stores the entered profile information and transmits it to the server. The input is user information, and the local storage and transmission process communicates this to the server as output.

[0189] Step 5:

[0190] The server stores the received information in a database and generates a profile based on individual characteristics. In this process, the entered user information is received and stored in a database. At this time, a profile is generated based on the stored data and stored in the server.

[0191] Step 6:

[0192] The user inputs daily schedules and tasks by voice or text. The input here is specific schedule and task information, such as "I have a meeting at 1:00 p.m." The output is generated as text data.

[0193] Step 7:

[0194] The device converts voice input into text data and sends that data to the server. Voice data is input, and text data is output through voice recognition. This text data is then sent to the server.

[0195] Step 8:

[0196] The server analyzes the received text data and generates an action plan. This analysis takes the text data as input, extracts key information using natural language processing, and outputs a notification plan. The generated action plan includes specific content, such as "send a reminder five minutes before the meeting and a notification to prepare materials."

[0197] Step 9:

[0198] The server sends the generated action plan to the terminal. The input here is the generated action plan, and the output is notification data sent to the terminal.

[0199] Step 10:

[0200] The terminal stores the action plan locally and displays it to the user, with the received action plan as input and the action plan stored locally and displayed on the user interface as output.

[0201] Step 11:

[0202] The server schedules notifications based on the generated action plan, where the input is the content of the action plan and the output is the schedule for each notification.

[0203] Step 12:

[0204] The server sends scheduled notification information to the terminal, which has the schedule information as input and sends it to the terminal as output.

[0205] Step 13:

[0206] The device triggers a reminder based on the schedule and displays a notification to the user. The input here is schedule information, and the output is a reminder that is triggered and notified to the user in real time.

[0207] Step 14:

[0208] When a user needs support in communicating with others, they input a specific example. For example, they input "the contents of a conversation with their boss." The input can be a conversation scenario, and text data is generated as the output.

[0209] Step 15:

[0210] The device converts the voice into text data and sends it to the server. The input is voice data, and the output is text data, which is then sent to the server.

[0211] Step 16:

[0212] The server analyzes the received text data and generates the optimal communication support message. The input is text data, and an appropriate message is output through natural language processing. For example, "Hello, boss. I have a request for you today."

[0213] Step 17:

[0214] The server generates and sends messages to the terminal. It has the generated message as input and sends it to the terminal as output.

[0215] Step 18:

[0216] The terminal displays the message to the user. The received message is the input, and the output is displayed on the user interface. The user can use this as a reference for communication.

[0217] (Application example 1)

[0218] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0219] There is a need to solve the problems that users with developmental disabilities face in their daily and social lives, and to realize smooth communication and efficient support between staff and customers with developmental disabilities, especially in brick-and-mortar stores. However, it is difficult for staff to instantly determine the appropriate response method based on the characteristics of each customer, which leads to a decrease in customer satisfaction and an increase in the burden on staff.

[0220] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0221] In this invention, the server includes means for accepting user input information, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, means for sending the notifications to a terminal, means for supporting interactions with customers with developmental disabilities, and means for displaying the customer interaction support information in real time. This enables store staff to instantly determine appropriate responses for customers with developmental disabilities and provide efficient support. Specifically, action plans and communication support messages tailored to the customer can be displayed in real time through the staff's smart glasses, enabling smooth customer interaction.

[0222] "User input information" refers to information about work or tasks that a user provides to the system.

[0223] An "action plan" is a plan that instructs the user on a series of tasks or procedures to be performed based on the input information.

[0224] "Notification" is a message that notifies the user of the generated action plan at an appropriate time.

[0225] A "terminal" is a device that receives information from the system, such as a smartphone, smart glasses, a head-mounted display, or a computer.

[0226] "Means to support the treatment of customers with developmental disabilities" refers to a system that provides support functions to enable staff to provide appropriate assistance to customers with developmental disabilities when they receive services in stores.

[0227] "Means for displaying customer support information in real time" is a function that instantly displays the information necessary when responding to a customer on the staff member's terminal.

[0228] "Profile Information" means information about a user's basic information and characteristics.

[0229] "Means for converting voice input into text data" refers to technology for converting voice input by a user into document data.

[0230] "Means for extracting appropriate actions" is a function that analyzes input data to find appropriate actions or instructions.

[0231] "Reminders" is a feature that notifies users so that they do not forget scheduled actions or events.

[0232] "Means for providing text and voice notifications" refers to a function that notifies users of action instructions and reminders via text message or voice.

[0233] The system required to implement this invention includes the following major components:

[0234] Generating a Program

[0235] The server accepts and analyzes the user's input, generates an action plan, schedules notifications based on the action plan, sends the notifications to the device, and displays information to support interactions with customers with developmental disabilities in real time.

[0236] Program processing explanation

[0237] Hardware and software:

[0238] Hardware: smart glasses, smartphones, computers

[0239] Software: API communication library (requests), speech synthesis library (text_to_speech), generative AI model

[0240] Data processing and calculation flow:

[0241] 1. Acceptance and analysis of user input:

[0242] Users input tasks and schedules through smart glasses or smartphones, using voice input or text input.

[0243] The device receives this input information, converts it to text data if it is voice input, and then analyzes the input data to extract key information (time, action items, etc.).

[0244] 2. Action plan generation and notification schedule:

[0245] The server receives the extracted data and uses a generative AI model to generate an action plan, including the necessary preparations and associated tasks.

[0246] The server schedules the timing of notifications based on the action plan and transmits notification information to the terminal.

[0247] 3. Real-time display of customer support:

[0248] When staff wear smart glasses, the information they need to assist customers is displayed in real time on the glasses' display, and voice notifications are also provided.

[0249] Support messages based on profile information are also provided to help facilitate smooth communication with customers.

[0250] Examples:

[0251] Example 1: Generating an action plan:

[0252] The user speaks, "I have a meeting to explain the new product at 2 p.m."

[0253] The device converts the speech into text and sends it to the server.

[0254] The server analyzes the received text data and generates an action plan (such as sending a reminder five minutes before a meeting or a notification to prepare materials).

[0255] Based on the action plan, the device will notify the employee at 1:55 p.m., "It's five minutes before the meeting. Please prepare your materials," and at 2:00 p.m. will send a reminder, "It's time for the meeting."

[0256] Example 2: Communication support:

[0257] The user enters the text "I have a question about the placement of a new product."

[0258] The terminal analyzes the received text and sends it to the server.

[0259] The server generates a communication support message ("You have a question about the product. What kind of placement would you like?") based on the analyzed data.

[0260] The terminal displays the generated message on the user's smart glasses to support communication with the customer.

[0261] Prompt Sentence Examples

[0262] Sample prompt: "What is the best way to respond to a customer with a developmental disability who asks about new product placement?"

[0263] Example prompt: "Please tell me what you've prepared for the 2pm meeting."

[0264] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0265] Step 1:

[0266] Users input tasks and schedules using smart glasses or a smartphone. Input can be done by voice or text. The device accepts this input and converts it into text data in the case of voice input.

[0267] Input: User voice or text input

[0268] Output: Text data

[0269] Specific operation: The user inputs "I have a meeting to explain the new product at 2 p.m." The device converts the voice into text and generates this text data.

[0270] Step 2:

[0271] The device analyzes the generated text data, extracts key information (task name, time, etc.), and sends it to the server.

[0272] Input: Text data

[0273] Output: Data containing key information

[0274] Specific operation: The terminal analyzes the text "There will be a meeting to explain the new product at 2 p.m.", extracts the key information "2 p.m." and "new product explanation meeting," and sends this to the server.

[0275] Step 3:

[0276] Based on the key information received, the server uses a generative AI model to generate an action plan, including necessary preparations and reminder settings.

[0277] Input: Data containing key information

[0278] Output: Action plan

[0279] Specific operation: Based on the key information about the "meeting to explain the new product at 2 p.m.," the server sets a reminder five minutes before the meeting and generates an action plan to prepare the materials.

[0280] Step 4:

[0281] The server schedules the timing of notifications based on the generated action plan and transmits the notification information to the terminal.

[0282] Input: Action Plan

[0283] Output: Notification schedule and notification information

[0284] Specific operation: Based on the action plan, the server schedules notifications such as "notify preparation of materials at 1:55 pm" and "remind the start of the meeting at 2 pm" and sends that information to the terminal.

[0285] Step 5:

[0286] The device will then notify the user at the specified time based on the notification information received. In the case of smart glasses, notifications are displayed in real time on the screen and, if necessary, are also made via voice.

[0287] Input: Notification information

[0288] Output: Notification to user

[0289] Specific operation: At 1:55 PM, the device displays a message on the smart glasses saying, "It's five minutes until the meeting. Please prepare your materials." It also issues a voice notification. At 2:00 PM, it notifies the user again, saying, "It's time for the meeting."

[0290] Step 6:

[0291] To support the user in communicating with the customer, the server generates a communication support message and transmits it to the terminal, which displays it in real time.

[0292] Input: Customer question or context

[0293] Output: Communication support message

[0294] Specific operation: The user inputs "I have a question about the placement of a new product." Based on this information, the server generates a communication support message saying, "You have a question about the product. What kind of placement would you like?" and sends it to the terminal. The terminal displays this message on the smart glasses in real time, supporting smooth communication with the customer.

[0295] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0296] MODE FOR CARRYING OUT THE INVENTION

[0297] System Overview

[0298] This invention is a support system for solving problems faced by users with developmental disabilities in their daily and social lives. This system accepts and analyzes the user's input information and emotional state, generates an action plan, and presents notifications at appropriate times, providing an environment in which the user can live more smoothly.

[0299] Handling of user profiles

[0300] server

[0301] When a user first launches an application, the server provides a screen for setting user profile information.

[0302] Users enter their personal information (e.g., name, age, characteristics of developmental disabilities, etc.), needs and support methods related to the characteristics, and emotional state.

[0303] Terminal

[0304] The device stores the entered profile information locally and sends it to the server.

[0305] The server stores the received information in a database and generates a profile based on the characteristics.

[0306] Generate an action plan

[0307] User

[0308] The user inputs daily schedules and tasks. This can be done by voice input or text input. For example, "I have a meeting at 1pm."

[0309] Terminal

[0310] The device converts voice input into text data and analyzes it to extract key information, which identifies specific times and action items.

[0311] server

[0312] It receives the analyzed text data and generates an action plan (pre-meeting preparation, reminders, related tasks, etc.) based on the user's characteristics and emotional state.

[0313] The action plan will include specific meeting times, materials to be presented, and necessary preparations.

[0314] Terminal

[0315] The generated action plan is sent to the device and stored locally, where the user can review it and adjust it if necessary.

[0316] Notifications and Reminders

[0317] server

[0318] Based on the generated action plan, schedule the timing of the notification (e.g., 5 minutes before the meeting).

[0319] The notification information is sent to the terminal.

[0320] Terminal

[0321] The reminder will be triggered at the scheduled time, and a notification will be displayed to the user via their device (smartphone, smart glasses, wireless earphones, etc.).

[0322] Communication Support

[0323] User

[0324] When a user needs assistance communicating with others, they can input a specific example (e.g., a conversation with their boss) using voice or text input.

[0325] Terminal

[0326] The device converts the voice into text data and analyzes it.

[0327] server

[0328] Based on the analyzed text data, the system generates an optimal communication support message for the user, such as "Hello, boss. I have a request for you today."

[0329] Terminal

[0330] The generated message is sent to the terminal and provided to the user, who can refer to it for smooth communication.

[0331] Recognizing and responding to emotional states

[0332] User

[0333] While the user is using the system, their emotional state is collected through a facial recognition camera and voice input.

[0334] Terminal

[0335] The collected facial recognition and voice data is analyzed in real time and input into the emotion engine.

[0336] server

[0337] The emotion engine analyzes the input data and identifies the user's current emotional state (happiness, anger, sadness, surprise, etc.).

[0338] The emotion engine generates and adjusts appropriate action plans and communication support messages based on the identified emotional state.

[0339] For example, if a user is feeling stressed, the system will flexibly adjust the timing of reminders and send notifications encouraging them to relax.

[0340] Terminal

[0341] Based on the recognized emotional state, appropriate notifications and support messages are sent to the device and provided to the user in real time.

[0342] Specific examples

[0343] Example 1: Generating an action plan

[0344] A user speaks, "I have a meeting at 1pm."

[0345] The device converts the speech into text and sends it to the server.

[0346] The server analyzes the received text data and generates an action plan.

[0347] The generated action plan includes a five-minute reminder before the meeting and notifications to prepare materials.

[0348] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[0349] Example 2: Communication support

[0350] A user enters the text "I have trouble talking to my boss."

[0351] The terminal analyzes the received text and sends it to the server.

[0352] The server generates a communication support message based on the parsed data.

[0353] The generated message is "Hello boss. I have a favor to ask of you today."

[0354] The terminal displays the generated message to the user to support communication.

[0355] Example 3: Recognizing and responding to emotional states

[0356] Users begin to feel stressed while using the system.

[0357] The device's facial recognition camera and voice input collect emotional states and transmit them to the emotion engine in real time.

[0358] The server's emotion engine recognizes when the user is feeling stressed and generates messages encouraging relaxation, such as "It's time to take a little break."

[0359] The terminal displays the generated message to the user and responds to the user's emotions.

[0360] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics and emotional state of the user.

[0361] The processing flow will be explained below.

[0362] User profile settings

[0363] Step 1:

[0364] User

[0365] Launch the application for the first time and access the profile information input screen.

[0366] Step 2:

[0367] User

[0368] Enter the name, age, characteristics of the developmental disorder (e.g., ADHD, ASD, LD, etc.), and support needs related to the characteristics.

[0369] Step 3:

[0370] Terminal

[0371] The entered profile information is temporarily stored in local storage.

[0372] Step 4:

[0373] Terminal

[0374] Send the information stored in the local storage to the server.

[0375] Step 5:

[0376] server

[0377] The received profile information is stored in a database and a profile based on the characteristics is generated.

[0378] Generate an action plan

[0379] Step 1:

[0380] User

[0381] Enter your daily schedule and tasks (e.g., I have a meeting at 1pm) using voice or text.

[0382] Step 2:

[0383] Terminal

[0384] Converts voice input into text data and analyzes it to extract key information (e.g., date and time, action items).

[0385] Step 3:

[0386] Terminal

[0387] The parsed text data is sent to the server.

[0388] Step 4:

[0389] server

[0390] Based on the received text data, it generates an action plan tailored to each individual's characteristics and needs, including a timeline, preparation list, reminders, and more.

[0391] Step 5:

[0392] server

[0393] The generated action plan is transmitted to the terminal.

[0394] Step 6:

[0395] Terminal

[0396] It stores the received action plan in local storage, notifies the user, and provides the user with the option to adjust the plan if necessary.

[0397] Notifications and Reminders

[0398] Step 1:

[0399] server

[0400] Based on your action plan, schedule notification timing for each action (e.g., a reminder 5 minutes before a meeting).

[0401] Step 2:

[0402] server

[0403] Scheduled notification information is sent to the terminal.

[0404] Step 3:

[0405] Terminal

[0406] Based on the notification timing, the action plan stored in the local storage is checked.

[0407] Step 4:

[0408] Terminal

[0409] A reminder is generated based on the notification timing, and a notification is displayed or sounded to the user via a smartphone, smart glasses, or wireless earphones (for example, at 12:55, a notification is sent saying, "It's five minutes before the meeting. Please prepare your materials.").

[0410] Communication Support

[0411] Step 1:

[0412] User

[0413] If you need help communicating with others (e.g., talking to your boss), enter it by voice or text.

[0414] Step 2:

[0415] Terminal

[0416] Voice input is converted into text data and analyzed to extract specific needs.

[0417] Step 3:

[0418] Terminal

[0419] The extracted needs are transmitted to the server.

[0420] Step 4:

[0421] server

[0422] Based on the received data, the system generates an optimal communication support message for the user. For example, it generates "Hello, boss. I have a request for you today."

[0423] Step 5:

[0424] server

[0425] Send the generated support message to the device.

[0426] Step 6:

[0427] Terminal

[0428] The received message is displayed to the user, and the user is supported to carry out smooth communication based on the message.

[0429] Recognizing and responding to emotional states

[0430] Step 1:

[0431] User

[0432] While using the system, users provide their emotional state through camera and voice input, in real time.

[0433] Step 2:

[0434] Terminal

[0435] The collected facial recognition data and voice data are input into the emotion engine and analysis begins.

[0436] Step 3:

[0437] server

[0438] The emotion engine analyzes facial recognition and voice data to identify the user's emotional state (e.g., joy, anger, sadness, surprise, etc.).

[0439] Step 4:

[0440] server

[0441] Adapt action plans and communication support messages based on the identified emotional state. For example, if the user is feeling stressed, flexibly change the timing of reminders and generate notifications such as "take time to relax."

[0442] Step 5:

[0443] server

[0444] Send coordinated action plans and notifications to your device.

[0445] Step 6:

[0446] Terminal

[0447] Display received notifications and support messages to users and provide appropriate support in real time.

[0448] Specific examples

[0449] Example 1: Generating an action plan

[0450] A user speaks, "I have a meeting at 1pm."

[0451] The device converts the speech into text and sends it to the server.

[0452] The server analyzes the received text data and generates an action plan.

[0453] The generated action plan includes a five-minute reminder before the meeting and notifications to prepare materials.

[0454] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[0455] Example 2: Communication support

[0456] A user enters the text "I have trouble talking to my boss."

[0457] The terminal analyzes the received text and sends it to the server.

[0458] The server generates a communication support message based on the parsed data.

[0459] The generated message is "Hello boss. I have a favor to ask of you today."

[0460] The terminal displays the generated message to the user to support communication.

[0461] Example 3: Recognizing and responding to emotional states

[0462] Users begin to feel stressed while using the system.

[0463] The device's facial recognition camera and voice input collect emotional states and transmit them to the emotion engine in real time.

[0464] The server's emotion engine recognizes when the user is feeling stressed and generates messages encouraging relaxation, such as "It's time to take a little break."

[0465] The terminal displays the generated message to the user and responds to the user's emotions.

[0466] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics and emotional state of the user.

[0467] Example 2

[0468] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0469] There is a need for systems that can efficiently resolve the challenges faced by users with developmental disabilities in their daily and social lives and support them in leading smoother lives. In particular, it is necessary to recognize fluctuations in emotional states and communication difficulties in real time and provide appropriate action plans and support. However, conventional systems have had difficulty fully meeting these requirements and have not been able to provide flexible and appropriate responses that meet the needs of users.

[0470] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0471] In this invention, the server includes means for accepting input information from a user, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, means for sending the notifications to a terminal, means for collecting emotional states through a face recognition camera or voice input, and means for analyzing the emotional states and taking appropriate measures, thereby enabling real-time behavioral and communication support according to the user's characteristics and current emotional state.

[0472] "Means for accepting user input information" refers to a function that provides an interface that allows users to input information into the system in voice, text, or other formats.

[0473] "Means for analyzing input information and generating action plans" refers to algorithms or processes that analyze the input information received from the user and automatically create specific action plans or tasks based on that information.

[0474] The "means for scheduling notifications based on an action plan" is a function for setting notifications at appropriate times according to the contents of the generated action plan.

[0475] "Means for sending notifications to devices" refers to the functionality that delivers scheduled notifications to the user's device (smartphone, tablet, etc.).

[0476] "Means of collecting emotional state through facial recognition cameras and voice input" refers to a system that acquires the user's facial expressions and voice data in real time and determines their emotions based on that.

[0477] "Means for analyzing emotional states and taking appropriate action" refers to the process of analyzing the acquired emotional data and generating an action plan or support message that matches the user's current mood and emotions.

[0478] "Means for setting user profile information" refers to a function that provides an interface that allows users to register and set their own basic information, characteristics, and needs.

[0479] "Means for storing profile information" refers to a function that securely stores the configured user information in a database or local storage.

[0480] "Means for providing support tailored to the user's characteristics" refers to the process of providing assistance and support tailored to individual needs and characteristics based on stored user information.

[0481] "Means for converting voice input into text data" refers to voice recognition technology that converts information input by a user into text data.

[0482] "Means for analyzing text data and extracting appropriate actions" refers to a text analysis algorithm for finding necessary actions and information from voice and text data.

[0483] The "means for generating a reminder based on an action" is a function for creating a reminder (e.g., a schedule notification) according to the extracted action.

[0484] "Means for notifying reminders" refers to a function for displaying set reminders on the user's device.

[0485] "Means for generating and providing messages that support communication with others" refers to the function by which the system automatically generates and provides appropriate sentences and messages to enable users to communicate smoothly.

[0486] This invention is a support system for users with developmental disabilities to solve problems in their daily and social lives. This system accepts input information and emotional state of the user, analyzes it, generates an action plan, and notifies the user at an appropriate time, providing an environment in which the user can live smoothly.

[0487] System Configuration

[0488] This system consists of three elements: a server, a terminal, and a user.

[0489] server

[0490] It provides a profile information setting screen and receives user input information.

[0491] The received information is stored in a database.

[0492] It generates an action plan based on the user's characteristics and emotional state and schedules appropriate notification timing.

[0493] Use an emotion engine to analyze emotional states in real time and take appropriate action.

[0494] Terminal

[0495] The information entered by the user is stored locally and sent to the server.

[0496] It converts voice input into text data and analyzes that data to extract key information.

[0497] Displaying action plans and notifications from the server to the user.

[0498] Emotional states are collected through facial recognition cameras and voice input, analyzed in real time, and sent to the emotion engine.

[0499] User

[0500] Launch the application and enter your profile information.

[0501] Enter daily schedules and tasks using voice or text.

[0502] Enter specific examples of communication with others.

[0503] Review and adjust support messages and action plans from the system, if necessary.

[0504] Hardware and Software

[0505] The specific hardware and software used to realize this system are as follows:

[0506] Hardware

[0507] Server: A central server with a high-performance database and analysis engine.

[0508] Devices: smartphones, tablets, smart glasses, wireless earphones, etc.

[0509] Facial recognition camera: A camera installed on the device.

[0510] Audio input device: The device's built-in microphone.

[0511] software

[0512] Profile settings screen: A web application or mobile application provided on the server side.

[0513] Speech Recognition System: Software for converting voice input into text data.

[0514] Text analysis engine: An AI model for analyzing text data and extracting key information.

[0515] Emotion analysis engine: An AI model that analyzes voice data and facial recognition data to identify emotional states.

[0516] Notification system: Software for scheduling and sending notifications to devices based on action plans.

[0517] Specific examples

[0518] Example 1: Generating an action plan

[0519] 1. The user says, "I have a meeting at 1:00 PM."

[0520] 2. The device converts the speech into text and sends it to the server.

[0521] 3. The server analyzes the received text data and generates an action plan, which includes a five-minute reminder before the meeting and notifications to prepare materials.

[0522] 4. At 12:55, the device will notify you, "It's 5 minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[0523] Example 2: Communication support

[0524] 1. The user types in the text "I have trouble talking to my boss."

[0525] 2. The device analyzes the received text and sends it to the server.

[0526] 3. The server generates a communication support message based on the analyzed data. The generated message is "Hello, boss. I have a request for you today."

[0527] 4. The terminal displays the generated message to the user to support communication.

[0528] Example 3: Recognizing and responding to emotional states

[0529] 1. Users begin to feel stressed while using the system.

[0530] 2. The device's facial recognition camera and voice input collect emotional states and transmit them to the emotion engine in real time.

[0531] 3. The server's emotion engine recognizes that the user is feeling stressed and generates a message encouraging relaxation, such as "It's time to take a short break."

[0532] 4. The device displays the generated message to the user and responds to the user's emotions.

[0533] Prompt Sentence Examples

[0534] "Please explain how users use this system to solve daily and social challenges related to developmental disabilities."

[0535] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics and emotional state of the user.

[0536] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0537] Step 1: Setting up your user profile

[0538] The server provides a user profile information setting screen when the application is first launched. The user enters personal information such as name, age, and characteristics of developmental disabilities. The device stores the entered information locally and sends it to the server. The server stores the received information in a database and generates a profile based on the user's characteristics.

[0539] Input: Personal information (name, age, characteristics of developmental disorders, etc.)

[0540] Data processing: saving and analyzing profile information

[0541] Output: Generate profile and save it to database

[0542] Step 2: Enter your daily schedule and tasks

[0543] Users input their daily schedules and tasks using voice or text. For example, they might input "I have a meeting at 1:00 PM." The device converts the voice input into text data, analyzes the data, extracts specific information (times and action items), and sends it to the server.

[0544] Input: Appointments and tasks (voice or text)

[0545] Data conversion: speech-to-text conversion, text data analysis

[0546] Output: Analysis results, specific times and action items

[0547] Step 3: Generate an action plan

[0548] The server generates an action plan based on the received text data, taking into account the user's characteristics and emotional state. The action plan includes meeting reminders and instructions for preparing materials. The action plan is then sent from the server to the device.

[0549] Input: Parsed text data, user characteristics information

[0550] Data Computation: Generating Action Plans

[0551] Output: Action plan

[0552] Step 4: Communicate and schedule your action plan

[0553] The server schedules the timing of notifications based on the generated action plan. The schedule information is sent to the device, and the device sets the time when the reminder will be sent. For example, at 12:55, the server sends a notification saying, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it sends a reminder saying, "It's time for the meeting."

[0554] Input: Action Plan

[0555] Data Processing: Notification Schedule Settings

[0556] Output: Schedule information, reminder notifications

[0557] Step 5: Support for communicating with others

[0558] When a user needs specific support regarding communication with others, they input a case (e.g., a conversation with their boss). The device converts the received voice into text and sends it to the server. The server analyzes the data, generates a communication support message, and sends it to the device. The device then displays the generated message to the user.

[0559] Input: Communication instance (audio or text)

[0560] Data conversion: speech-to-text conversion, text data analysis

[0561] Output: Support message

[0562] Step 6: Recognize and respond to emotional states

[0563] The system collects the user's emotional state while using the system. The device's facial recognition camera and voice input device send data to the emotion engine in real time. The server's emotion engine analyzes the input data and identifies the user's emotional state. If the server recognizes a specific emotion, such as stress, it generates an appropriate response message and sends it to the device. For example, it displays a message encouraging relaxation, such as "It's time to take a short break."

[0564] Input: Face recognition data, voice input data

[0565] Data analysis: Emotional state analysis

[0566] Output: Response message, adjustment of action plan

[0567] (Application example 2)

[0568] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0569] This invention relates to a support system for solving problems faced by users with developmental disabilities in their daily and social lives. Current technologies are inadequate in providing timely reminders and support messages to users who have difficulties shopping in physical stores or using services. Furthermore, there is a lack of technology that recognizes emotional states and provides appropriate support based on those states in real time. As a result, users are not adequately supported in their daily and social lives.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0571] In this invention, the server includes means for accepting user input information, means for analyzing the input information and generating an action plan, means for recognizing the user's emotional state, means for generating reminders and support messages based on the emotional state, and means for generating messages to support communication with others. This allows users with developmental disabilities to receive appropriate reminders and support messages in a timely manner when shopping or using services at physical stores. In addition, appropriate support based on the user's emotional state is provided in real time, and communication support is also provided, thereby reducing the user's stress and difficulties and enabling them to live a smooth daily and social life.

[0572] "User" refers to an individual with a developmental disability who uses the System.

[0573] "Input information" refers to information that a user provides to the system through voice input, text input, or the like.

[0574] An "action plan" is a system-generated plan of specific actions and reminders based on the specific tasks or schedules a user faces.

[0575] "Notifications" are alerts or messages sent to the device at times or in situations specified by the system based on the action plan.

[0576] "Terminal" means a device for receiving notifications from the system, including a smartphone, smart glasses, a head-mounted display, and other electronic devices.

[0577] "Profile Information" means information about you, including personal information such as your name, age, and developmental disability characteristics.

[0578] "Support" refers to assistance including action plans, reminders, support messages, and communication support messages provided according to the user's characteristics and emotional state.

[0579] "Emotional state" refers to the emotions (e.g., joy, anger, sadness, surprise, stress, etc.) that users express while using the system.

[0580] "Emotion recognition" is a technology that identifies a user's emotional state in real time based on data collected from their voice input and facial recognition cameras.

[0581] A "communication support message" is a specific guide message generated by the system to help users communicate smoothly with others.

[0582] A "physical store" is a commercial facility or service provider that has a physical location, where customers visit to purchase goods or use services.

[0583] The present invention describes a specific embodiment of a support system for enabling users with developmental disabilities to smoothly shop and use services in brick-and-mortar stores. This system includes a server and a terminal, and is configured as follows.

[0584] System Overview

[0585] server

[0586] The server includes means for accepting input information from a user, means for analyzing the input information and generating an action plan, means for recognizing the emotional state of the user, means for generating reminders and support messages based on the emotional state, and means for generating messages to support communication with others.

[0587] Terminal

[0588] The devices include smartphones, smart glasses, head-mounted displays, and other electronic devices that receive notifications from the server and display them on their screens. The devices also use cameras and microphones to collect the user's emotional state and send it to the server. Specific software used includes EmotionRecognizer (an emotion recognition engine) and TextToSpeech (a voice synthesis engine).

[0589] Operational Overview

[0590] Profile setup and support

[0591] When a user launches the application for the first time, the server provides a user profile setting screen where the user enters personal information such as name, age, and characteristics, which the device stores locally and then sends to the server.

[0592] The server stores the received information in a database and generates a profile based on the user's characteristics. Based on this profile, the server provides support tailored to the user's characteristics.

[0593] Action plan generation and notification

[0594] When a user inputs their plans for the store (for example, purchasing products or checking in at a service counter) by voice or text, the terminal converts this into text data and sends it to the server.

[0595] The server generates an action plan and related reminders based on the analyzed text data and sends them to the device.

[0596] The device will display a reminder at the specified time, prompting the user to take action.

[0597] Recognizing and responding to emotional states

[0598] The device uses a camera and microphone to collect the user's facial expressions and voice, and inputs this into an emotion recognition engine in real time.

[0599] The server's emotion recognition engine analyzes the user's current emotional state and generates appropriate reminder and support messages based on this. For example, if the user is feeling stressed, it will display a message such as "It's time to take a short break."

[0600] Communication Support

[0601] When a user needs to communicate with others, the user inputs a specific example (for example, the content of a conversation with a store clerk) by voice or text.

[0602] The device analyzes the text data and sends it to the server. The server generates an optimal communication support message based on the analysis results and sends it to the device.

[0603] The terminal displays the generated message to the user, thereby supporting smooth communication.

[0604] Examples and Prompts

[0605] Usage example

[0606] For example, if a user plans to "purchase a product at 1:00 p.m.", the following steps are taken:

[0607] The user speaks, "I'll buy the product at 1pm."

[0608] The terminal converts this into text and sends it to the server.

[0609] The server generates an action plan and generates reminders such as "Start preparing to buy the product at 12:55."

[0610] The device will display a reminder at 12:55, prompting the user to take action.

[0611] If you feel stressed while making a purchase, the device will recognize your emotions in real time and tell you, "It's time to take a break."

[0612] Prompt Sentence Examples

[0613] "Design an application to help customers with developmental disabilities easily shop in brick-and-mortar stores. Use the smartphone's camera and microphone to recognize their emotional state in real time and provide timely reminders and supportive messages."

[0614] As described above, a support system that enables users with developmental disabilities to smoothly shop and use services in physical stores can be realized using a configuration that uses a server and terminals.

[0615] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0616] Step 1:

[0617] When a user launches the application for the first time, they enter their profile settings screen. They enter their personal information, such as their name, age, and characteristics. The device stores this locally and sends it to the server. The entered information is then stored in a database as personal information data.

[0618] Step 2:

[0619] The server analyzes the received user profile information and generates a profile to provide support based on the user's characteristics. The profile includes the user's characteristics and the support items they need, and is then sent to the device. Database search and analysis algorithms are used to generate the profile.

[0620] Step 3:

[0621] The user inputs their plans for the physical store (e.g., "Purchase the product at 1:00 PM") into the system using voice or text. The device converts the voice into text data and sends it to the server. The converted text data is saved as input plan data.

[0622] Step 4:

[0623] The server analyzes the received text data and generates an action plan and reminders using natural language processing and scheduling algorithms. The action plan, which includes specific tasks and their timing, is then sent to the device.

[0624] Step 5:

[0625] The server sets a reminder notification schedule based on the generated action plan. The schedule setting includes adding a timestamp and saving it in an event list. The set schedule is then sent to the device as notification data.

[0626] Step 6:

[0627] The device displays a reminder at a specified time to prompt the user to take action. For example, it displays a message saying, "Please start preparing to purchase the product at 12:55." This can be displayed as a pop-up notification or a sound notification. This triggers movement or action in the physical store.

[0628] Step 7:

[0629] The device uses a camera and microphone to collect the user's emotional state and inputs it into the EmotionRecognizer (emotion recognition engine) in real time. The collected data is saved as facial expression data and voice data.

[0630] Step 8:

[0631] The server uses an emotion recognition engine to analyze the collected facial expression and voice data to identify the user's current emotional state. This analysis uses machine learning algorithms, and the identification result is stored on the server as emotional state data.

[0632] Step 9:

[0633] The server generates appropriate reminders and support messages based on the identified emotional state. For example, if the user is feeling stressed, it generates a message such as "It's time to take a short break." The generated message is then sent to the device as support data.

[0634] Step 10:

[0635] When a user needs support in communicating with others, they input a specific example by voice or text. The device converts this into text data and sends it to the server. The converted text data is saved as communication input data.

[0636] Step 11:

[0637] The server analyzes the received communication input data and generates an optimal communication support message using natural language processing and generative AI models. The generated message is then sent to the device as communication support data.

[0638] Step 12:

[0639] The terminal displays the generated communication support message to the user to support smooth communication. For example, it displays "Hello, store clerk. I need your help." This display completes the support for the user to communicate appropriately.

[0640] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0641] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0642] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0643] [Second embodiment]

[0644] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0645] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0646] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0647] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0648] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0649] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0650] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0651] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0652] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0653] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0654] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0655] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0656] MODE FOR CARRYING OUT THE INVENTION

[0657] System Overview

[0658] This invention is a support system for solving problems faced by users with developmental disabilities in their daily and social lives. This system accepts and analyzes user input, generates an action plan, and presents notifications at appropriate times, providing an environment in which users can live more smoothly.

[0659] Handling of user profiles

[0660] server

[0661] When a user first launches an application, the server provides a screen for setting user profile information.

[0662] Users enter their personal information (e.g., name, age, characteristics of developmental disabilities, etc.), needs related to those characteristics, and support methods.

[0663] Terminal

[0664] The device stores the entered profile information locally and sends it to the server.

[0665] The server stores the received information in a database and generates a profile based on individual characteristics.

[0666] Generate an action plan

[0667] User

[0668] The user inputs daily schedules and tasks. This can be done by voice input or text input. For example, "I have a meeting at 1pm."

[0669] Terminal

[0670] The device converts voice input into text data and analyzes it to extract key information, which identifies specific times and action items.

[0671] server

[0672] It receives the analyzed text data and generates an action plan (pre-meeting preparation, reminders, related tasks, etc.) based on the user's characteristics and needs.

[0673] The action plan will include specific meeting times, materials to be presented, and necessary preparations.

[0674] Terminal

[0675] The generated action plan is sent to the device and stored locally, where the user can review it and adjust it if necessary.

[0676] Notifications and Reminders

[0677] server

[0678] Based on the generated action plan, schedule the timing of the notification (e.g., 5 minutes before the meeting).

[0679] The notification information is sent to the terminal.

[0680] Terminal

[0681] The reminder will be triggered at the scheduled time, and a notification will be displayed to the user via their device (smartphone, smart glasses, wireless earphones, etc.).

[0682] Communication Support

[0683] User

[0684] When a user needs assistance communicating with others, they can input a specific example (e.g., a conversation with their boss) using voice or text input.

[0685] Terminal

[0686] The device converts the voice into text data and analyzes it.

[0687] server

[0688] Based on the analyzed text data, the system generates an optimal communication support message for the user, such as "Hello, boss. I have a request for you today."

[0689] Terminal

[0690] The generated message is sent to the terminal and provided to the user, who can refer to it for smooth communication.

[0691] Specific examples

[0692] Example 1: Generating an action plan

[0693] A user speaks, "I have a meeting at 1pm."

[0694] The device converts the speech into text and sends it to the server.

[0695] The server analyzes the received text data and generates an action plan (such as sending a reminder five minutes before a meeting or a notification to prepare materials).

[0696] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[0697] Example 2: Communication support

[0698] A user enters the text "I have trouble talking to my boss."

[0699] The terminal analyzes the received text and sends it to the server.

[0700] The server generates a communication support message ("Hello, boss. I have a request for you today.") based on the analyzed data.

[0701] The terminal displays the generated message to the user to support communication.

[0702] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics of the user.

[0703] The processing flow will be explained below.

[0704] User profile settings

[0705] Step 1:

[0706] User The user launches the application for the first time and accesses the profile information input screen.

[0707] Step 2:

[0708] User Enter name, age, characteristics of developmental disorder (e.g., ADHD, ASD, LD, etc.), and support needs related to the characteristics.

[0709] Step 3:

[0710] Terminal The entered information is temporarily saved in the terminal's local storage.

[0711] Step 4:

[0712] The information stored in the device's local storage is sent to the server.

[0713] Step 5:

[0714] Server: Stores the received profile information in a database and generates a profile based on the characteristics.

[0715] Generate an action plan

[0716] Step 1:

[0717] User: Enter daily schedules and tasks (e.g., I have a meeting at 1:00 p.m.) using voice or text.

[0718] Step 2:

[0719] Device Converts voice input into text data and analyzes it to extract key information (e.g., date and time, action items).

[0720] Step 3:

[0721] The terminal sends the analyzed text data to the server.

[0722] Step 4:

[0723] Server: Based on the received text data, it generates an action plan tailored to each individual's characteristics and needs, including a timeline, preparation list, reminders, etc.

[0724] Step 5:

[0725] Server: Sends the generated action plan to the terminal.

[0726] Step 6:

[0727] The device stores the received action plan in local storage, notifies the user, and provides the user with the option to adjust the plan if necessary.

[0728] Notifications and Reminders

[0729] Step 1:

[0730] Server Based on the action plan, schedule the notification timing for each action (e.g., a reminder 5 minutes before a meeting).

[0731] Step 2:

[0732] Server: Sends scheduled notification information to the terminal.

[0733] Step 3:

[0734] Device Check the action plan stored in local storage based on the notification timing.

[0735] Step 4:

[0736] Device: A reminder is generated based on the notification timing, and a notification is displayed or audible to the user via a smartphone, smart glasses, or wireless earphones (for example, at 12:55, a notification is sent saying, "It's five minutes before the meeting. Please prepare your materials.").

[0737] Communication Support

[0738] Step 1:

[0739] User When there is something that requires support in communicating with others (e.g., a conversation with a boss), the user inputs it by voice or text.

[0740] Step 2:

[0741] The device converts voice input into text data, analyzes it, and extracts specific needs.

[0742] Step 3:

[0743] The terminal transmits the extracted needs to the server.

[0744] Step 4:

[0745] Server: Based on the received data, it generates the most appropriate communication support message for the user (e.g., "Hello, boss. I have a request for you today.").

[0746] Step 5:

[0747] The server sends the generated support message to the terminal.

[0748] Step 6:

[0749] Terminal: Displays received messages to the user, helping the user to communicate smoothly based on the messages.

[0750] Example 1

[0751] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0752] Users with developmental disabilities face challenges in their daily and social lives, including schedule management and smooth communication. These challenges can cause difficulties in daily and social activities if users do not receive appropriate support. The present invention aims to provide an effective solution to these challenges.

[0753] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0754] In this invention, the server includes means for accepting user input information, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, means for locally saving the generated action plan, and means for displaying notifications to the user. This enables an action plan to be automatically generated based on information input by the user and notifications to be sent at appropriate times. The server also includes means for setting and saving user profile information and means for providing support tailored to the user's characteristics. This enables detailed support tailored to the characteristics of each individual user. The server also includes means for converting voice input into text data, means for analyzing the text data and extracting appropriate actions, and means for generating a communication support message optimized for the user. This also enables smooth communication support through voice input.

[0755] "User" refers to an individual with a developmental disability who uses the system.

[0756] "Input information" refers to data that a user provides to the system, and includes, for example, schedules, tasks, profile information, and the like.

[0757] "Analysis" refers to the processing of data based on input information to understand the content and generate relevant action plans and notifications.

[0758] An "action plan" is a list of schedules and tasks generated based on input information, and is intended to support users' daily and social lives.

[0759] "Notification" means information provided to a user at an appropriate time based on an action plan, including reminders and alerts.

[0760] "Schedule" refers to the time and order in which notifications are set based on an action plan.

[0761] A "terminal" is a hardware device for running the system, including smartphones and tablets.

[0762] "Local storage" refers to the temporary or long-term storage of data within a device.

[0763] "Profile information" refers to personal information provided by the user to the system, including name, age, and characteristics of the developmental disorder.

[0764] "Support" refers to providing assistance tailored to the characteristics and needs of users.

[0765] "Voice input" refers to the means by which a user provides information to a system using speech.

[0766] "Text data" refers to data obtained by converting voice input into text information.

[0767] "Communication support messages" are messages generated based on analyzed data to help users communicate smoothly with others.

[0768] This invention is a system for solving problems faced by users with developmental disabilities in their daily and social lives. This system accepts and analyzes information input by the user, generates an action plan, and presents notifications at appropriate times, thereby providing an environment in which the user can live more smoothly.

[0769] Configuration and Operation Procedures

[0770] 1. Handling of User Profiles

[0771] When a user first launches the application, the server provides a user profile information setup screen, where the user enters their name, age, characteristics of their developmental disorder, support needs, etc. The device then stores the entered profile information locally and sends it to the server. The server then stores the received information in a database and generates a profile based on each individual's characteristics.

[0772] 2. Generate an action plan

[0773] The user inputs daily schedules and tasks using voice or text. For example, "I have a meeting at 1 p.m." The device converts the voice input into text data and analyzes it to extract key information. This analysis identifies specific times and action items. The server receives the analyzed text data and generates an action plan (pre-meeting preparations, reminders, and related tasks) based on the user's characteristics and needs. The generated action plan may include, for example, "send a reminder 5 minutes before the meeting and a notification to prepare materials." The action plan is then sent to the device and stored locally. The user can review it and adjust it as needed.

[0774] 3. Notifications and Reminders

[0775] The server schedules the notification based on the generated action plan. For example, it sets a reminder five minutes before a meeting. The notification information is sent to the device. The device triggers the reminder at the scheduled time, and a notification is displayed to the user via their smartphone or other device.

[0776] 4. Communication support

[0777] When a user needs support in communicating with others, they input a specific example (e.g., the contents of a conversation with their boss). This is done by voice input or text input. The device converts the voice into text data and analyzes it. The server generates an optimal communication support message for the user based on the analyzed text data. For example, it generates a specific message such as, "Hello, boss. I have a request for you today." The device displays the generated message to the user to support communication.

[0778] Specific examples

[0779] Example 1: Generating an action plan

[0780] A user speaks, "I have a meeting at 1pm."

[0781] The device converts the speech into text and sends it to the server.

[0782] The server analyzes the received text data and generates an action plan (such as sending a reminder five minutes before a meeting or a notification to prepare materials).

[0783] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[0784] Example 2: Communication support

[0785] A user enters the text "I have trouble talking to my boss."

[0786] The terminal analyzes the received text and sends it to the server.

[0787] The server generates a communication support message ("Hello, boss. I have a request for you today.") based on the analyzed data.

[0788] The terminal displays the generated message to the user to support communication.

[0789] Prompt Sentence Examples

[0790] "Please explain in natural language the process of dictating that there will be a meeting at 1pm."

[0791] "Please explain in detail the process for generating support messages for users who are uncomfortable speaking with their managers."

[0792] Through these steps, the system of the present invention provides an environment in which users with developmental disabilities can overcome challenges in their daily and social lives and live more smoothly.

[0793] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0794] Step 1:

[0795] The user initially launches the application, which starts the system operation and provides an input that displays a profile setting screen for the user.

[0796] Step 2:

[0797] The server provides a user profile information setup screen, allowing the user to input their name, age, characteristics of their developmental disorder, support needs, etc. This input leads to an output that captures the user's basic personal information.

[0798] Step 3:

[0799] The user enters personal information such as name, age, and characteristics of the developmental disorder. This input provides the system with basic data to provide support tailored to the user's characteristics. This data is then used in later steps to generate an action plan and support messages tailored to the user's characteristics.

[0800] Step 4:

[0801] The device locally stores the entered profile information and transmits it to the server. The input is user information, and the local storage and transmission process communicates this to the server as output.

[0802] Step 5:

[0803] The server stores the received information in a database and generates a profile based on individual characteristics. In this process, the entered user information is received and stored in a database. At this time, a profile is generated based on the stored data and stored in the server.

[0804] Step 6:

[0805] The user inputs daily schedules and tasks by voice or text. The input here is specific schedule and task information, such as "I have a meeting at 1:00 p.m." The output is generated as text data.

[0806] Step 7:

[0807] The device converts voice input into text data and sends that data to the server. Voice data is input, and text data is output through voice recognition. This text data is then sent to the server.

[0808] Step 8:

[0809] The server analyzes the received text data and generates an action plan. This analysis takes the text data as input, extracts key information using natural language processing, and outputs a notification plan. The generated action plan includes specific content, such as "send a reminder five minutes before the meeting and a notification to prepare materials."

[0810] Step 9:

[0811] The server sends the generated action plan to the terminal. The input here is the generated action plan, and the output is notification data sent to the terminal.

[0812] Step 10:

[0813] The terminal stores the action plan locally and displays it to the user, with the received action plan as input and the action plan stored locally and displayed on the user interface as output.

[0814] Step 11:

[0815] The server schedules notifications based on the generated action plan, where the input is the content of the action plan and the output is the schedule for each notification.

[0816] Step 12:

[0817] The server sends scheduled notification information to the terminal, which has the schedule information as input and sends it to the terminal as output.

[0818] Step 13:

[0819] The device triggers a reminder based on the schedule and displays a notification to the user. The input here is schedule information, and the output is a reminder that is triggered and notified to the user in real time.

[0820] Step 14:

[0821] When a user needs support in communicating with others, they input a specific example. For example, they input "the contents of a conversation with their boss." The input can be a conversation scenario, and text data is generated as the output.

[0822] Step 15:

[0823] The device converts the voice into text data and sends it to the server. The input is voice data, and the output is text data, which is then sent to the server.

[0824] Step 16:

[0825] The server analyzes the received text data and generates the optimal communication support message. The input is text data, and an appropriate message is output through natural language processing. For example, "Hello, boss. I have a request for you today."

[0826] Step 17:

[0827] The server generates and sends messages to the terminal. It has the generated message as input and sends it to the terminal as output.

[0828] Step 18:

[0829] The terminal displays the message to the user. The received message is the input, and the output is displayed on the user interface. The user can use this as a reference for communication.

[0830] (Application example 1)

[0831] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0832] There is a need to solve the problems that users with developmental disabilities face in their daily and social lives, and to realize smooth communication and efficient support between staff and customers with developmental disabilities, especially in brick-and-mortar stores. However, it is difficult for staff to instantly determine the appropriate response method based on the characteristics of each customer, which leads to a decrease in customer satisfaction and an increase in the burden on staff.

[0833] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0834] In this invention, the server includes means for accepting user input information, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, means for sending the notifications to a terminal, means for supporting interactions with customers with developmental disabilities, and means for displaying the customer interaction support information in real time. This enables store staff to instantly determine appropriate responses for customers with developmental disabilities and provide efficient support. Specifically, action plans and communication support messages tailored to the customer can be displayed in real time through the staff's smart glasses, enabling smooth customer interaction.

[0835] "User input information" refers to information about work or tasks that a user provides to the system.

[0836] An "action plan" is a plan that instructs the user on a series of tasks or procedures to be performed based on the input information.

[0837] "Notification" is a message that notifies the user of the generated action plan at an appropriate time.

[0838] A "terminal" is a device that receives information from the system, such as a smartphone, smart glasses, a head-mounted display, or a computer.

[0839] "Means to support the treatment of customers with developmental disabilities" refers to a system that provides support functions to enable staff to provide appropriate assistance to customers with developmental disabilities when they receive services in stores.

[0840] "Means for displaying customer support information in real time" is a function that instantly displays the information necessary when responding to a customer on the staff member's terminal.

[0841] "Profile Information" means information about a user's basic information and characteristics.

[0842] "Means for converting voice input into text data" refers to technology for converting voice input by a user into document data.

[0843] "Means for extracting appropriate actions" is a function that analyzes input data to find appropriate actions or instructions.

[0844] "Reminders" is a feature that notifies users so that they do not forget scheduled actions or events.

[0845] "Means for providing text and voice notifications" refers to a function that notifies users of action instructions and reminders via text message or voice.

[0846] The system required to implement this invention includes the following major components:

[0847] Generating a Program

[0848] The server accepts and analyzes the user's input, generates an action plan, schedules notifications based on the action plan, sends the notifications to the device, and displays information to support interactions with customers with developmental disabilities in real time.

[0849] Program processing explanation

[0850] Hardware and software:

[0851] Hardware: smart glasses, smartphones, computers

[0852] Software: API communication library (requests), speech synthesis library (text_to_speech), generative AI model

[0853] Data processing and calculation flow:

[0854] 1. Acceptance and analysis of user input:

[0855] Users input tasks and schedules through smart glasses or smartphones, using voice input or text input.

[0856] The device receives this input information, converts it to text data if it is voice input, and then analyzes the input data to extract key information (time, action items, etc.).

[0857] 2. Action plan generation and notification schedule:

[0858] The server receives the extracted data and uses a generative AI model to generate an action plan, including the necessary preparations and associated tasks.

[0859] The server schedules the timing of notifications based on the action plan and transmits notification information to the terminal.

[0860] 3. Real-time display of customer support:

[0861] When staff wear smart glasses, the information they need to assist customers is displayed in real time on the glasses' display, and voice notifications are also provided.

[0862] Support messages based on profile information are also provided to help facilitate smooth communication with customers.

[0863] Examples:

[0864] Example 1: Generating an action plan:

[0865] The user speaks, "I have a meeting to explain the new product at 2 p.m."

[0866] The device converts the speech into text and sends it to the server.

[0867] The server analyzes the received text data and generates an action plan (such as sending a reminder five minutes before a meeting or a notification to prepare materials).

[0868] Based on the action plan, the device will notify the employee at 1:55 p.m., "It's five minutes before the meeting. Please prepare your materials," and at 2:00 p.m. will send a reminder, "It's time for the meeting."

[0869] Example 2: Communication support:

[0870] The user enters the text "I have a question about the placement of a new product."

[0871] The terminal analyzes the received text and sends it to the server.

[0872] The server generates a communication support message ("You have a question about the product. What kind of placement would you like?") based on the analyzed data.

[0873] The terminal displays the generated message on the user's smart glasses to support communication with the customer.

[0874] Prompt Sentence Examples

[0875] Sample prompt: "What is the best way to respond to a customer with a developmental disability who asks about new product placement?"

[0876] Example prompt: "Please tell me what you've prepared for the 2pm meeting."

[0877] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0878] Step 1:

[0879] Users input tasks and schedules using smart glasses or a smartphone. Input can be done by voice or text. The device accepts this input and converts it into text data in the case of voice input.

[0880] Input: User voice or text input

[0881] Output: Text data

[0882] Specific operation: The user inputs "I have a meeting to explain the new product at 2 p.m." The device converts the voice into text and generates this text data.

[0883] Step 2:

[0884] The device analyzes the generated text data, extracts key information (task name, time, etc.), and sends it to the server.

[0885] Input: Text data

[0886] Output: Data containing key information

[0887] Specific operation: The terminal analyzes the text "There will be a meeting to explain the new product at 2 p.m.", extracts the key information "2 p.m." and "new product explanation meeting," and sends this to the server.

[0888] Step 3:

[0889] Based on the key information received, the server uses a generative AI model to generate an action plan, including necessary preparations and reminder settings.

[0890] Input: Data containing key information

[0891] Output: Action plan

[0892] Specific operation: Based on the key information about the "meeting to explain the new product at 2 p.m.," the server sets a reminder five minutes before the meeting and generates an action plan to prepare the materials.

[0893] Step 4:

[0894] The server schedules the timing of notifications based on the generated action plan and transmits the notification information to the terminal.

[0895] Input: Action Plan

[0896] Output: Notification schedule and notification information

[0897] Specific operation: Based on the action plan, the server schedules notifications such as "notify preparation of materials at 1:55 pm" and "remind the start of the meeting at 2 pm" and sends that information to the terminal.

[0898] Step 5:

[0899] The device will then notify the user at the specified time based on the notification information received. In the case of smart glasses, notifications are displayed in real time on the screen and, if necessary, are also made via voice.

[0900] Input: Notification information

[0901] Output: Notification to user

[0902] Specific operation: At 1:55 PM, the device displays a message on the smart glasses saying, "It's five minutes until the meeting. Please prepare your materials." It also issues a voice notification. At 2:00 PM, it notifies the user again, saying, "It's time for the meeting."

[0903] Step 6:

[0904] To support the user in communicating with the customer, the server generates a communication support message and transmits it to the terminal, which displays it in real time.

[0905] Input: Customer question or context

[0906] Output: Communication support message

[0907] Specific operation: The user inputs "I have a question about the placement of a new product." Based on this information, the server generates a communication support message saying, "You have a question about the product. What kind of placement would you like?" and sends it to the terminal. The terminal displays this message on the smart glasses in real time, supporting smooth communication with the customer.

[0908] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0909] MODE FOR CARRYING OUT THE INVENTION

[0910] System Overview

[0911] This invention is a support system for solving problems faced by users with developmental disabilities in their daily and social lives. This system accepts and analyzes the user's input information and emotional state, generates an action plan, and presents notifications at appropriate times, providing an environment in which the user can live more smoothly.

[0912] Handling of user profiles

[0913] server

[0914] When a user first launches an application, the server provides a screen for setting user profile information.

[0915] Users enter their personal information (e.g., name, age, characteristics of developmental disabilities, etc.), needs and support methods related to the characteristics, and emotional state.

[0916] Terminal

[0917] The device stores the entered profile information locally and sends it to the server.

[0918] The server stores the received information in a database and generates a profile based on the characteristics.

[0919] Generate an action plan

[0920] User

[0921] The user inputs daily schedules and tasks. This can be done by voice input or text input. For example, "I have a meeting at 1pm."

[0922] Terminal

[0923] The device converts voice input into text data and analyzes it to extract key information, which identifies specific times and action items.

[0924] server

[0925] It receives the analyzed text data and generates an action plan (pre-meeting preparation, reminders, related tasks, etc.) based on the user's characteristics and emotional state.

[0926] The action plan will include specific meeting times, materials to be presented, and necessary preparations.

[0927] Terminal

[0928] The generated action plan is sent to the device and stored locally, where the user can review it and adjust it if necessary.

[0929] Notifications and Reminders

[0930] server

[0931] Based on the generated action plan, schedule the timing of the notification (e.g., 5 minutes before the meeting).

[0932] The notification information is sent to the terminal.

[0933] Terminal

[0934] The reminder will be triggered at the scheduled time, and a notification will be displayed to the user via their device (smartphone, smart glasses, wireless earphones, etc.).

[0935] Communication Support

[0936] User

[0937] When a user needs assistance communicating with others, they can input a specific example (e.g., a conversation with their boss) using voice or text input.

[0938] Terminal

[0939] The device converts the voice into text data and analyzes it.

[0940] server

[0941] Based on the analyzed text data, the system generates an optimal communication support message for the user, such as "Hello, boss. I have a request for you today."

[0942] Terminal

[0943] The generated message is sent to the terminal and provided to the user, who can refer to it for smooth communication.

[0944] Recognizing and responding to emotional states

[0945] User

[0946] While the user is using the system, their emotional state is collected through a facial recognition camera and voice input.

[0947] Terminal

[0948] The collected facial recognition and voice data is analyzed in real time and input into the emotion engine.

[0949] server

[0950] The emotion engine analyzes the input data and identifies the user's current emotional state (happiness, anger, sadness, surprise, etc.).

[0951] The emotion engine generates and adjusts appropriate action plans and communication support messages based on the identified emotional state.

[0952] For example, if a user is feeling stressed, the system will flexibly adjust the timing of reminders and send notifications encouraging them to relax.

[0953] Terminal

[0954] Based on the recognized emotional state, appropriate notifications and support messages are sent to the device and provided to the user in real time.

[0955] Specific examples

[0956] Example 1: Generating an action plan

[0957] A user speaks, "I have a meeting at 1pm."

[0958] The device converts the speech into text and sends it to the server.

[0959] The server analyzes the received text data and generates an action plan.

[0960] The generated action plan includes a five-minute reminder before the meeting and notifications to prepare materials.

[0961] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[0962] Example 2: Communication support

[0963] A user enters the text "I have trouble talking to my boss."

[0964] The terminal analyzes the received text and sends it to the server.

[0965] The server generates a communication support message based on the parsed data.

[0966] The generated message is "Hello boss. I have a favor to ask of you today."

[0967] The terminal displays the generated message to the user to support communication.

[0968] Example 3: Recognizing and responding to emotional states

[0969] Users begin to feel stressed while using the system.

[0970] The device's facial recognition camera and voice input collect emotional states and transmit them to the emotion engine in real time.

[0971] The server's emotion engine recognizes when the user is feeling stressed and generates messages encouraging relaxation, such as "It's time to take a little break."

[0972] The terminal displays the generated message to the user and responds to the user's emotions.

[0973] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics and emotional state of the user.

[0974] The processing flow will be explained below.

[0975] User profile settings

[0976] Step 1:

[0977] User

[0978] Launch the application for the first time and access the profile information input screen.

[0979] Step 2:

[0980] User

[0981] Enter the name, age, characteristics of the developmental disorder (e.g., ADHD, ASD, LD, etc.), and support needs related to the characteristics.

[0982] Step 3:

[0983] Terminal

[0984] The entered profile information is temporarily stored in local storage.

[0985] Step 4:

[0986] Terminal

[0987] Send the information stored in the local storage to the server.

[0988] Step 5:

[0989] server

[0990] The received profile information is stored in a database and a profile based on the characteristics is generated.

[0991] Generate an action plan

[0992] Step 1:

[0993] User

[0994] Enter your daily schedule and tasks (e.g., I have a meeting at 1pm) using voice or text.

[0995] Step 2:

[0996] Terminal

[0997] Converts voice input into text data and analyzes it to extract key information (e.g., date and time, action items).

[0998] Step 3:

[0999] Terminal

[1000] The parsed text data is sent to the server.

[1001] Step 4:

[1002] server

[1003] Based on the received text data, it generates an action plan tailored to each individual's characteristics and needs, including a timeline, preparation list, reminders, and more.

[1004] Step 5:

[1005] server

[1006] The generated action plan is transmitted to the terminal.

[1007] Step 6:

[1008] Terminal

[1009] It stores the received action plan in local storage, notifies the user, and provides the user with the option to adjust the plan if necessary.

[1010] Notifications and Reminders

[1011] Step 1:

[1012] server

[1013] Based on your action plan, schedule notification timing for each action (e.g., a reminder 5 minutes before a meeting).

[1014] Step 2:

[1015] server

[1016] Scheduled notification information is sent to the terminal.

[1017] Step 3:

[1018] Terminal

[1019] Based on the notification timing, the action plan stored in the local storage is checked.

[1020] Step 4:

[1021] Terminal

[1022] A reminder is generated based on the notification timing, and a notification is displayed or sounded to the user via a smartphone, smart glasses, or wireless earphones (for example, at 12:55, a notification is sent saying, "It's five minutes before the meeting. Please prepare your materials.").

[1023] Communication Support

[1024] Step 1:

[1025] User

[1026] If you need help communicating with others (e.g., talking to your boss), enter it by voice or text.

[1027] Step 2:

[1028] Terminal

[1029] Voice input is converted into text data and analyzed to extract specific needs.

[1030] Step 3:

[1031] Terminal

[1032] The extracted needs are transmitted to the server.

[1033] Step 4:

[1034] server

[1035] Based on the received data, the system generates an optimal communication support message for the user. For example, it generates "Hello, boss. I have a request for you today."

[1036] Step 5:

[1037] server

[1038] Send the generated support message to the device.

[1039] Step 6:

[1040] Terminal

[1041] The received message is displayed to the user, and the user is supported to carry out smooth communication based on the message.

[1042] Recognizing and responding to emotional states

[1043] Step 1:

[1044] User

[1045] While using the system, users provide their emotional state through camera and voice input, in real time.

[1046] Step 2:

[1047] Terminal

[1048] The collected facial recognition data and voice data are input into the emotion engine and analysis begins.

[1049] Step 3:

[1050] server

[1051] The emotion engine analyzes facial recognition and voice data to identify the user's emotional state (e.g., joy, anger, sadness, surprise, etc.).

[1052] Step 4:

[1053] server

[1054] Adapt action plans and communication support messages based on the identified emotional state. For example, if the user is feeling stressed, flexibly change the timing of reminders and generate notifications such as "take time to relax."

[1055] Step 5:

[1056] server

[1057] Send coordinated action plans and notifications to your device.

[1058] Step 6:

[1059] Terminal

[1060] Display received notifications and support messages to users and provide appropriate support in real time.

[1061] Specific examples

[1062] Example 1: Generating an action plan

[1063] A user speaks, "I have a meeting at 1pm."

[1064] The device converts the speech into text and sends it to the server.

[1065] The server analyzes the received text data and generates an action plan.

[1066] The generated action plan includes a five-minute reminder before the meeting and notifications to prepare materials.

[1067] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[1068] Example 2: Communication support

[1069] A user enters the text "I have trouble talking to my boss."

[1070] The terminal analyzes the received text and sends it to the server.

[1071] The server generates a communication support message based on the parsed data.

[1072] The generated message is "Hello boss. I have a favor to ask of you today."

[1073] The terminal displays the generated message to the user to support communication.

[1074] Example 3: Recognizing and responding to emotional states

[1075] Users begin to feel stressed while using the system.

[1076] The device's facial recognition camera and voice input collect emotional states and transmit them to the emotion engine in real time.

[1077] The server's emotion engine recognizes when the user is feeling stressed and generates messages encouraging relaxation, such as "It's time to take a little break."

[1078] The terminal displays the generated message to the user and responds to the user's emotions.

[1079] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics and emotional state of the user.

[1080] Example 2

[1081] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1082] There is a need for systems that can efficiently resolve the challenges faced by users with developmental disabilities in their daily and social lives and support them in leading smoother lives. In particular, it is necessary to recognize fluctuations in emotional states and communication difficulties in real time and provide appropriate action plans and support. However, conventional systems have had difficulty fully meeting these requirements and have not been able to provide flexible and appropriate responses that meet the needs of users.

[1083] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1084] In this invention, the server includes means for accepting input information from a user, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, means for sending the notifications to a terminal, means for collecting emotional states through a face recognition camera or voice input, and means for analyzing the emotional states and taking appropriate measures, thereby enabling real-time behavioral and communication support according to the user's characteristics and current emotional state.

[1085] "Means for accepting user input information" refers to a function that provides an interface that allows users to input information into the system in voice, text, or other formats.

[1086] "Means for analyzing input information and generating action plans" refers to algorithms or processes that analyze the input information received from the user and automatically create specific action plans or tasks based on that information.

[1087] The "means for scheduling notifications based on an action plan" is a function for setting notifications at appropriate times according to the contents of the generated action plan.

[1088] "Means for sending notifications to devices" refers to the functionality that delivers scheduled notifications to the user's device (smartphone, tablet, etc.).

[1089] "Means of collecting emotional state through facial recognition cameras and voice input" refers to a system that acquires the user's facial expressions and voice data in real time and determines their emotions based on that.

[1090] "Means for analyzing emotional states and taking appropriate action" refers to the process of analyzing the acquired emotional data and generating an action plan or support message that matches the user's current mood and emotions.

[1091] "Means for setting user profile information" refers to a function that provides an interface that allows users to register and set their own basic information, characteristics, and needs.

[1092] "Means for storing profile information" refers to a function that securely stores the configured user information in a database or local storage.

[1093] "Means for providing support tailored to the user's characteristics" refers to the process of providing assistance and support tailored to individual needs and characteristics based on stored user information.

[1094] "Means for converting voice input into text data" refers to voice recognition technology that converts information input by a user into text data.

[1095] "Means for analyzing text data and extracting appropriate actions" refers to a text analysis algorithm for finding necessary actions and information from voice and text data.

[1096] The "means for generating a reminder based on an action" is a function for creating a reminder (e.g., a schedule notification) according to the extracted action.

[1097] "Means for notifying reminders" refers to a function for displaying set reminders on the user's device.

[1098] "Means for generating and providing messages that support communication with others" refers to the function by which the system automatically generates and provides appropriate sentences and messages to enable users to communicate smoothly.

[1099] This invention is a support system for users with developmental disabilities to solve problems in their daily and social lives. This system accepts input information and emotional state of the user, analyzes it, generates an action plan, and notifies the user at an appropriate time, providing an environment in which the user can live smoothly.

[1100] System Configuration

[1101] This system consists of three elements: a server, a terminal, and a user.

[1102] server

[1103] It provides a profile information setting screen and receives user input information.

[1104] The received information is stored in a database.

[1105] It generates an action plan based on the user's characteristics and emotional state and schedules appropriate notification timing.

[1106] Use an emotion engine to analyze emotional states in real time and take appropriate action.

[1107] Terminal

[1108] The information entered by the user is stored locally and sent to the server.

[1109] It converts voice input into text data and analyzes that data to extract key information.

[1110] Displaying action plans and notifications from the server to the user.

[1111] Emotional states are collected through facial recognition cameras and voice input, analyzed in real time, and sent to the emotion engine.

[1112] User

[1113] Launch the application and enter your profile information.

[1114] Enter daily schedules and tasks using voice or text.

[1115] Enter specific examples of communication with others.

[1116] Review and adjust support messages and action plans from the system, if necessary.

[1117] Hardware and Software

[1118] The specific hardware and software used to realize this system are as follows:

[1119] Hardware

[1120] Server: A central server with a high-performance database and analysis engine.

[1121] Devices: smartphones, tablets, smart glasses, wireless earphones, etc.

[1122] Facial recognition camera: A camera installed on the device.

[1123] Audio input device: The device's built-in microphone.

[1124] software

[1125] Profile settings screen: A web application or mobile application provided on the server side.

[1126] Speech Recognition System: Software for converting voice input into text data.

[1127] Text analysis engine: An AI model for analyzing text data and extracting key information.

[1128] Emotion analysis engine: An AI model that analyzes voice data and facial recognition data to identify emotional states.

[1129] Notification system: Software for scheduling and sending notifications to devices based on action plans.

[1130] Specific examples

[1131] Example 1: Generating an action plan

[1132] 1. The user says, "I have a meeting at 1:00 PM."

[1133] 2. The device converts the speech into text and sends it to the server.

[1134] 3. The server analyzes the received text data and generates an action plan, which includes a five-minute reminder before the meeting and notifications to prepare materials.

[1135] 4. At 12:55, the device will notify you, "It's 5 minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[1136] Example 2: Communication support

[1137] 1. The user types in the text "I have trouble talking to my boss."

[1138] 2. The device analyzes the received text and sends it to the server.

[1139] 3. The server generates a communication support message based on the analyzed data. The generated message is "Hello, boss. I have a request for you today."

[1140] 4. The terminal displays the generated message to the user to support communication.

[1141] Example 3: Recognizing and responding to emotional states

[1142] 1. Users begin to feel stressed while using the system.

[1143] 2. The device's facial recognition camera and voice input collect emotional states and transmit them to the emotion engine in real time.

[1144] 3. The server's emotion engine recognizes that the user is feeling stressed and generates a message encouraging relaxation, such as "It's time to take a short break."

[1145] 4. The device displays the generated message to the user and responds to the user's emotions.

[1146] Prompt Sentence Examples

[1147] "Please explain how users use this system to solve daily and social challenges related to developmental disabilities."

[1148] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics and emotional state of the user.

[1149] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1150] Step 1: Setting up your user profile

[1151] The server provides a user profile information setting screen when the application is first launched. The user enters personal information such as name, age, and characteristics of developmental disabilities. The device stores the entered information locally and sends it to the server. The server stores the received information in a database and generates a profile based on the user's characteristics.

[1152] Input: Personal information (name, age, characteristics of developmental disorders, etc.)

[1153] Data processing: saving and analyzing profile information

[1154] Output: Generate profile and save it to database

[1155] Step 2: Enter your daily schedule and tasks

[1156] Users input their daily schedules and tasks using voice or text. For example, they might input "I have a meeting at 1:00 PM." The device converts the voice input into text data, analyzes the data, extracts specific information (times and action items), and sends it to the server.

[1157] Input: Appointments and tasks (voice or text)

[1158] Data conversion: speech-to-text conversion, text data analysis

[1159] Output: Analysis results, specific times and action items

[1160] Step 3: Generate an action plan

[1161] The server generates an action plan based on the received text data, taking into account the user's characteristics and emotional state. The action plan includes meeting reminders and instructions for preparing materials. The action plan is then sent from the server to the device.

[1162] Input: Parsed text data, user characteristics information

[1163] Data Computation: Generating Action Plans

[1164] Output: Action plan

[1165] Step 4: Communicate and schedule your action plan

[1166] The server schedules the timing of notifications based on the generated action plan. The schedule information is sent to the device, and the device sets the time when the reminder will be sent. For example, at 12:55, the server sends a notification saying, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it sends a reminder saying, "It's time for the meeting."

[1167] Input: Action Plan

[1168] Data Processing: Notification Schedule Settings

[1169] Output: Schedule information, reminder notifications

[1170] Step 5: Support for communicating with others

[1171] When a user needs specific support regarding communication with others, they input a case (e.g., a conversation with their boss). The device converts the received voice into text and sends it to the server. The server analyzes the data, generates a communication support message, and sends it to the device. The device then displays the generated message to the user.

[1172] Input: Communication instance (audio or text)

[1173] Data conversion: speech-to-text conversion, text data analysis

[1174] Output: Support message

[1175] Step 6: Recognize and respond to emotional states

[1176] The system collects the user's emotional state while using the system. The device's facial recognition camera and voice input device send data to the emotion engine in real time. The server's emotion engine analyzes the input data and identifies the user's emotional state. If the server recognizes a specific emotion, such as stress, it generates an appropriate response message and sends it to the device. For example, it displays a message encouraging relaxation, such as "It's time to take a short break."

[1177] Input: Face recognition data, voice input data

[1178] Data analysis: Emotional state analysis

[1179] Output: Response message, adjustment of action plan

[1180] (Application example 2)

[1181] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1182] This invention relates to a support system for solving problems faced by users with developmental disabilities in their daily and social lives. Current technologies are inadequate in providing timely reminders and support messages to users who have difficulties shopping in physical stores or using services. Furthermore, there is a lack of technology that recognizes emotional states and provides appropriate support based on those states in real time. As a result, users are not adequately supported in their daily and social lives.

[1183] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1184] In this invention, the server includes means for accepting user input information, means for analyzing the input information and generating an action plan, means for recognizing the user's emotional state, means for generating reminders and support messages based on the emotional state, and means for generating messages to support communication with others. This allows users with developmental disabilities to receive appropriate reminders and support messages in a timely manner when shopping or using services at physical stores. In addition, appropriate support based on the user's emotional state is provided in real time, and communication support is also provided, thereby reducing the user's stress and difficulties and enabling them to live a smooth daily and social life.

[1185] "User" refers to an individual with a developmental disability who uses the System.

[1186] "Input information" refers to information that a user provides to the system through voice input, text input, or the like.

[1187] An "action plan" is a system-generated plan of specific actions and reminders based on the specific tasks or schedules a user faces.

[1188] "Notifications" are alerts or messages sent to the device at times or in situations specified by the system based on the action plan.

[1189] "Terminal" means a device for receiving notifications from the system, including a smartphone, smart glasses, a head-mounted display, and other electronic devices.

[1190] "Profile Information" means information about you, including personal information such as your name, age, and developmental disability characteristics.

[1191] "Support" refers to assistance including action plans, reminders, support messages, and communication support messages provided according to the user's characteristics and emotional state.

[1192] "Emotional state" refers to the emotions (e.g., joy, anger, sadness, surprise, stress, etc.) that users express while using the system.

[1193] "Emotion recognition" is a technology that identifies a user's emotional state in real time based on data collected from their voice input and facial recognition cameras.

[1194] A "communication support message" is a specific guide message generated by the system to help users communicate smoothly with others.

[1195] A "physical store" is a commercial facility or service provider that has a physical location, where customers visit to purchase goods or use services.

[1196] The present invention describes a specific embodiment of a support system for enabling users with developmental disabilities to smoothly shop and use services in brick-and-mortar stores. This system includes a server and a terminal, and is configured as follows.

[1197] System Overview

[1198] server

[1199] The server includes means for accepting input information from a user, means for analyzing the input information and generating an action plan, means for recognizing the emotional state of the user, means for generating reminders and support messages based on the emotional state, and means for generating messages to support communication with others.

[1200] Terminal

[1201] The devices include smartphones, smart glasses, head-mounted displays, and other electronic devices that receive notifications from the server and display them on their screens. The devices also use cameras and microphones to collect the user's emotional state and send it to the server. Specific software used includes EmotionRecognizer (an emotion recognition engine) and TextToSpeech (a voice synthesis engine).

[1202] Operational Overview

[1203] Profile setup and support

[1204] When a user launches the application for the first time, the server provides a user profile setting screen where the user enters personal information such as name, age, and characteristics, which the device stores locally and then sends to the server.

[1205] The server stores the received information in a database and generates a profile based on the user's characteristics. Based on this profile, the server provides support tailored to the user's characteristics.

[1206] Action plan generation and notification

[1207] When a user inputs their plans for the store (for example, purchasing products or checking in at a service counter) by voice or text, the terminal converts this into text data and sends it to the server.

[1208] The server generates an action plan and related reminders based on the analyzed text data and sends them to the device.

[1209] The device will display a reminder at the specified time, prompting the user to take action.

[1210] Recognizing and responding to emotional states

[1211] The device uses a camera and microphone to collect the user's facial expressions and voice, and inputs this into an emotion recognition engine in real time.

[1212] The server's emotion recognition engine analyzes the user's current emotional state and generates appropriate reminder and support messages based on this. For example, if the user is feeling stressed, it will display a message such as "It's time to take a short break."

[1213] Communication Support

[1214] When a user needs to communicate with others, the user inputs a specific example (for example, the content of a conversation with a store clerk) by voice or text.

[1215] The device analyzes the text data and sends it to the server. The server generates an optimal communication support message based on the analysis results and sends it to the device.

[1216] The terminal displays the generated message to the user, thereby supporting smooth communication.

[1217] Examples and Prompts

[1218] Usage example

[1219] For example, if a user plans to "purchase a product at 1:00 p.m.", the following steps are taken:

[1220] The user speaks, "I'll buy the product at 1pm."

[1221] The terminal converts this into text and sends it to the server.

[1222] The server generates an action plan and generates reminders such as "Start preparing to buy the product at 12:55."

[1223] The device will display a reminder at 12:55, prompting the user to take action.

[1224] If you feel stressed while making a purchase, the device will recognize your emotions in real time and tell you, "It's time to take a break."

[1225] Prompt Sentence Examples

[1226] "Design an application to help customers with developmental disabilities easily shop in brick-and-mortar stores. Use the smartphone's camera and microphone to recognize their emotional state in real time and provide timely reminders and supportive messages."

[1227] As described above, a support system that enables users with developmental disabilities to smoothly shop and use services in physical stores can be realized using a configuration that uses a server and terminals.

[1228] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1229] Step 1:

[1230] When a user launches the application for the first time, they enter their profile settings screen. They enter their personal information, such as their name, age, and characteristics. The device stores this locally and sends it to the server. The entered information is then stored in a database as personal information data.

[1231] Step 2:

[1232] The server analyzes the received user profile information and generates a profile to provide support based on the user's characteristics. The profile includes the user's characteristics and the support items they need, and is then sent to the device. Database search and analysis algorithms are used to generate the profile.

[1233] Step 3:

[1234] The user inputs their plans for the physical store (e.g., "Purchase the product at 1:00 PM") into the system using voice or text. The device converts the voice into text data and sends it to the server. The converted text data is saved as input plan data.

[1235] Step 4:

[1236] The server analyzes the received text data and generates an action plan and reminders using natural language processing and scheduling algorithms. The action plan, which includes specific tasks and their timing, is then sent to the device.

[1237] Step 5:

[1238] The server sets a reminder notification schedule based on the generated action plan. The schedule setting includes adding a timestamp and saving it in an event list. The set schedule is then sent to the device as notification data.

[1239] Step 6:

[1240] The device displays a reminder at a specified time to prompt the user to take action. For example, it displays a message saying, "Please start preparing to purchase the product at 12:55." This can be displayed as a pop-up notification or a sound notification. This triggers movement or action in the physical store.

[1241] Step 7:

[1242] The device uses a camera and microphone to collect the user's emotional state and inputs it into the EmotionRecognizer (emotion recognition engine) in real time. The collected data is saved as facial expression data and voice data.

[1243] Step 8:

[1244] The server uses an emotion recognition engine to analyze the collected facial expression and voice data to identify the user's current emotional state. This analysis uses machine learning algorithms, and the identification result is stored on the server as emotional state data.

[1245] Step 9:

[1246] The server generates appropriate reminders and support messages based on the identified emotional state. For example, if the user is feeling stressed, it generates a message such as "It's time to take a short break." The generated message is then sent to the device as support data.

[1247] Step 10:

[1248] When a user needs support in communicating with others, they input a specific example by voice or text. The device converts this into text data and sends it to the server. The converted text data is saved as communication input data.

[1249] Step 11:

[1250] The server analyzes the received communication input data and generates an optimal communication support message using natural language processing and generative AI models. The generated message is then sent to the device as communication support data.

[1251] Step 12:

[1252] The terminal displays the generated communication support message to the user to support smooth communication. For example, it displays "Hello, store clerk. I need your help." This display completes the support for the user to communicate appropriately.

[1253] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1254] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1255] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1256] [Third embodiment]

[1257] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1258] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1259] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1260] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1261] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1262] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1263] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1264] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1265] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1266] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1267] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1268] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1269] MODE FOR CARRYING OUT THE INVENTION

[1270] System Overview

[1271] This invention is a support system for solving problems faced by users with developmental disabilities in their daily and social lives. This system accepts and analyzes user input, generates an action plan, and presents notifications at appropriate times, providing an environment in which users can live more smoothly.

[1272] Handling of user profiles

[1273] server

[1274] When a user first launches an application, the server provides a screen for setting user profile information.

[1275] Users enter their personal information (e.g., name, age, characteristics of developmental disabilities, etc.), needs related to those characteristics, and support methods.

[1276] Terminal

[1277] The device stores the entered profile information locally and sends it to the server.

[1278] The server stores the received information in a database and generates a profile based on individual characteristics.

[1279] Generate an action plan

[1280] User

[1281] The user inputs daily schedules and tasks. This can be done by voice input or text input. For example, "I have a meeting at 1pm."

[1282] Terminal

[1283] The device converts voice input into text data and analyzes it to extract key information, which identifies specific times and action items.

[1284] server

[1285] It receives the analyzed text data and generates an action plan (pre-meeting preparation, reminders, related tasks, etc.) based on the user's characteristics and needs.

[1286] The action plan will include specific meeting times, materials to be presented, and necessary preparations.

[1287] Terminal

[1288] The generated action plan is sent to the device and stored locally, where the user can review it and adjust it if necessary.

[1289] Notifications and Reminders

[1290] server

[1291] Based on the generated action plan, schedule the timing of the notification (e.g., 5 minutes before the meeting).

[1292] The notification information is sent to the terminal.

[1293] Terminal

[1294] The reminder will be triggered at the scheduled time, and a notification will be displayed to the user via their device (smartphone, smart glasses, wireless earphones, etc.).

[1295] Communication Support

[1296] User

[1297] When a user needs assistance communicating with others, they can input a specific example (e.g., a conversation with their boss) using voice or text input.

[1298] Terminal

[1299] The device converts the voice into text data and analyzes it.

[1300] server

[1301] Based on the analyzed text data, the system generates an optimal communication support message for the user, such as "Hello, boss. I have a request for you today."

[1302] Terminal

[1303] The generated message is sent to the terminal and provided to the user, who can refer to it for smooth communication.

[1304] Specific examples

[1305] Example 1: Generating an action plan

[1306] A user speaks, "I have a meeting at 1pm."

[1307] The device converts the speech into text and sends it to the server.

[1308] The server analyzes the received text data and generates an action plan (such as sending a reminder five minutes before a meeting or a notification to prepare materials).

[1309] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[1310] Example 2: Communication support

[1311] A user enters the text "I have trouble talking to my boss."

[1312] The terminal analyzes the received text and sends it to the server.

[1313] The server generates a communication support message ("Hello, boss. I have a request for you today.") based on the analyzed data.

[1314] The terminal displays the generated message to the user to support communication.

[1315] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics of the user.

[1316] The processing flow will be explained below.

[1317] User profile settings

[1318] Step 1:

[1319] User The user launches the application for the first time and accesses the profile information input screen.

[1320] Step 2:

[1321] User Enter name, age, characteristics of developmental disorder (e.g., ADHD, ASD, LD, etc.), and support needs related to the characteristics.

[1322] Step 3:

[1323] Terminal The entered information is temporarily saved in the terminal's local storage.

[1324] Step 4:

[1325] The information stored in the device's local storage is sent to the server.

[1326] Step 5:

[1327] Server: Stores the received profile information in a database and generates a profile based on the characteristics.

[1328] Generate an action plan

[1329] Step 1:

[1330] User: Enter daily schedules and tasks (e.g., I have a meeting at 1:00 p.m.) using voice or text.

[1331] Step 2:

[1332] Device Converts voice input into text data and analyzes it to extract key information (e.g., date and time, action items).

[1333] Step 3:

[1334] The terminal sends the analyzed text data to the server.

[1335] Step 4:

[1336] Server: Based on the received text data, it generates an action plan tailored to each individual's characteristics and needs, including a timeline, preparation list, reminders, etc.

[1337] Step 5:

[1338] Server: Sends the generated action plan to the terminal.

[1339] Step 6:

[1340] The device stores the received action plan in local storage, notifies the user, and provides the user with the option to adjust the plan if necessary.

[1341] Notifications and Reminders

[1342] Step 1:

[1343] Server Based on the action plan, schedule the notification timing for each action (e.g., a reminder 5 minutes before a meeting).

[1344] Step 2:

[1345] Server: Sends scheduled notification information to the terminal.

[1346] Step 3:

[1347] Device Check the action plan stored in local storage based on the notification timing.

[1348] Step 4:

[1349] Device: A reminder is generated based on the notification timing, and a notification is displayed or audible to the user via a smartphone, smart glasses, or wireless earphones (for example, at 12:55, a notification is sent saying, "It's five minutes before the meeting. Please prepare your materials.").

[1350] Communication Support

[1351] Step 1:

[1352] User When there is something that requires support in communicating with others (e.g., a conversation with a boss), the user inputs it by voice or text.

[1353] Step 2:

[1354] The device converts voice input into text data, analyzes it, and extracts specific needs.

[1355] Step 3:

[1356] The terminal transmits the extracted needs to the server.

[1357] Step 4:

[1358] Server: Based on the received data, it generates the most appropriate communication support message for the user (e.g., "Hello, boss. I have a request for you today.").

[1359] Step 5:

[1360] The server sends the generated support message to the terminal.

[1361] Step 6:

[1362] Terminal: Displays received messages to the user, helping the user to communicate smoothly based on the messages.

[1363] Example 1

[1364] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1365] Users with developmental disabilities face challenges in their daily and social lives, including schedule management and smooth communication. These challenges can cause difficulties in daily and social activities if users do not receive appropriate support. The present invention aims to provide an effective solution to these challenges.

[1366] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1367] In this invention, the server includes means for accepting user input information, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, means for locally saving the generated action plan, and means for displaying notifications to the user. This enables an action plan to be automatically generated based on information input by the user and notifications to be sent at appropriate times. The server also includes means for setting and saving user profile information and means for providing support tailored to the user's characteristics. This enables detailed support tailored to the characteristics of each individual user. The server also includes means for converting voice input into text data, means for analyzing the text data and extracting appropriate actions, and means for generating a communication support message optimized for the user. This also enables smooth communication support through voice input.

[1368] "User" refers to an individual with a developmental disability who uses the system.

[1369] "Input information" refers to data that a user provides to the system, and includes, for example, schedules, tasks, profile information, and the like.

[1370] "Analysis" refers to the processing of data based on input information to understand the content and generate relevant action plans and notifications.

[1371] An "action plan" is a list of schedules and tasks generated based on input information, and is intended to support users' daily and social lives.

[1372] "Notification" means information provided to a user at an appropriate time based on an action plan, including reminders and alerts.

[1373] "Schedule" refers to the time and order in which notifications are set based on an action plan.

[1374] A "terminal" is a hardware device for running the system, including smartphones and tablets.

[1375] "Local storage" refers to the temporary or long-term storage of data within a device.

[1376] "Profile information" refers to personal information provided by the user to the system, including name, age, and characteristics of the developmental disorder.

[1377] "Support" refers to providing assistance tailored to the characteristics and needs of users.

[1378] "Voice input" refers to the means by which a user provides information to a system using speech.

[1379] "Text data" refers to data obtained by converting voice input into text information.

[1380] "Communication support messages" are messages generated based on analyzed data to help users communicate smoothly with others.

[1381] This invention is a system for solving problems faced by users with developmental disabilities in their daily and social lives. This system accepts and analyzes information input by the user, generates an action plan, and presents notifications at appropriate times, thereby providing an environment in which the user can live more smoothly.

[1382] Configuration and Operation Procedures

[1383] 1. Handling of User Profiles

[1384] When a user first launches the application, the server provides a user profile information setup screen, where the user enters their name, age, characteristics of their developmental disorder, support needs, etc. The device then stores the entered profile information locally and sends it to the server. The server then stores the received information in a database and generates a profile based on each individual's characteristics.

[1385] 2. Generate an action plan

[1386] The user inputs daily schedules and tasks using voice or text. For example, "I have a meeting at 1 p.m." The device converts the voice input into text data and analyzes it to extract key information. This analysis identifies specific times and action items. The server receives the analyzed text data and generates an action plan (pre-meeting preparations, reminders, and related tasks) based on the user's characteristics and needs. The generated action plan may include, for example, "send a reminder 5 minutes before the meeting and a notification to prepare materials." The action plan is then sent to the device and stored locally. The user can review it and adjust it as needed.

[1387] 3. Notifications and Reminders

[1388] The server schedules the notification based on the generated action plan. For example, it sets a reminder five minutes before a meeting. The notification information is sent to the device. The device triggers the reminder at the scheduled time, and a notification is displayed to the user via their smartphone or other device.

[1389] 4. Communication support

[1390] When a user needs support in communicating with others, they input a specific example (e.g., the contents of a conversation with their boss). This is done by voice input or text input. The device converts the voice into text data and analyzes it. The server generates an optimal communication support message for the user based on the analyzed text data. For example, it generates a specific message such as, "Hello, boss. I have a request for you today." The device displays the generated message to the user to support communication.

[1391] Specific examples

[1392] Example 1: Generating an action plan

[1393] A user speaks, "I have a meeting at 1pm."

[1394] The device converts the speech into text and sends it to the server.

[1395] The server analyzes the received text data and generates an action plan (such as sending a reminder five minutes before a meeting or a notification to prepare materials).

[1396] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[1397] Example 2: Communication support

[1398] A user enters the text "I have trouble talking to my boss."

[1399] The terminal analyzes the received text and sends it to the server.

[1400] The server generates a communication support message ("Hello, boss. I have a request for you today.") based on the analyzed data.

[1401] The terminal displays the generated message to the user to support communication.

[1402] Prompt Sentence Examples

[1403] "Please explain in natural language the process of dictating that there will be a meeting at 1pm."

[1404] "Please explain in detail the process for generating support messages for users who are uncomfortable speaking with their managers."

[1405] Through these steps, the system of the present invention provides an environment in which users with developmental disabilities can overcome challenges in their daily and social lives and live more smoothly.

[1406] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1407] Step 1:

[1408] The user initially launches the application, which starts the system operation and provides an input that displays a profile setting screen for the user.

[1409] Step 2:

[1410] The server provides a user profile information setup screen, allowing the user to input their name, age, characteristics of their developmental disorder, support needs, etc. This input leads to an output that captures the user's basic personal information.

[1411] Step 3:

[1412] The user enters personal information such as name, age, and characteristics of the developmental disorder. This input provides the system with basic data to provide support tailored to the user's characteristics. This data is then used in later steps to generate an action plan and support messages tailored to the user's characteristics.

[1413] Step 4:

[1414] The device locally stores the entered profile information and transmits it to the server. The input is user information, and the local storage and transmission process communicates this to the server as output.

[1415] Step 5:

[1416] The server stores the received information in a database and generates a profile based on individual characteristics. In this process, the entered user information is received and stored in a database. At this time, a profile is generated based on the stored data and stored in the server.

[1417] Step 6:

[1418] The user inputs daily schedules and tasks by voice or text. The input here is specific schedule and task information, such as "I have a meeting at 1:00 p.m." The output is generated as text data.

[1419] Step 7:

[1420] The device converts voice input into text data and sends that data to the server. Voice data is input, and text data is output through voice recognition. This text data is then sent to the server.

[1421] Step 8:

[1422] The server analyzes the received text data and generates an action plan. This analysis takes the text data as input, extracts key information using natural language processing, and outputs a notification plan. The generated action plan includes specific content, such as "send a reminder five minutes before the meeting and a notification to prepare materials."

[1423] Step 9:

[1424] The server sends the generated action plan to the terminal. The input here is the generated action plan, and the output is notification data sent to the terminal.

[1425] Step 10:

[1426] The terminal stores the action plan locally and displays it to the user, with the received action plan as input and the action plan stored locally and displayed on the user interface as output.

[1427] Step 11:

[1428] The server schedules notifications based on the generated action plan, where the input is the content of the action plan and the output is the schedule for each notification.

[1429] Step 12:

[1430] The server sends scheduled notification information to the terminal, which has the schedule information as input and sends it to the terminal as output.

[1431] Step 13:

[1432] The device triggers a reminder based on the schedule and displays a notification to the user. The input here is schedule information, and the output is a reminder that is triggered and notified to the user in real time.

[1433] Step 14:

[1434] When a user needs support in communicating with others, they input a specific example. For example, they input "the contents of a conversation with their boss." The input can be a conversation scenario, and text data is generated as the output.

[1435] Step 15:

[1436] The device converts the voice into text data and sends it to the server. The input is voice data, and the output is text data, which is then sent to the server.

[1437] Step 16:

[1438] The server analyzes the received text data and generates the optimal communication support message. The input is text data, and an appropriate message is output through natural language processing. For example, "Hello, boss. I have a request for you today."

[1439] Step 17:

[1440] The server generates and sends messages to the terminal. It has the generated message as input and sends it to the terminal as output.

[1441] Step 18:

[1442] The terminal displays the message to the user. The received message is the input, and the output is displayed on the user interface. The user can use this as a reference for communication.

[1443] (Application example 1)

[1444] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1445] There is a need to solve the problems that users with developmental disabilities face in their daily and social lives, and to realize smooth communication and efficient support between staff and customers with developmental disabilities, especially in brick-and-mortar stores. However, it is difficult for staff to instantly determine the appropriate response method based on the characteristics of each customer, which leads to a decrease in customer satisfaction and an increase in the burden on staff.

[1446] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1447] In this invention, the server includes means for accepting user input information, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, means for sending the notifications to a terminal, means for supporting interactions with customers with developmental disabilities, and means for displaying the customer interaction support information in real time. This enables store staff to instantly determine appropriate responses for customers with developmental disabilities and provide efficient support. Specifically, action plans and communication support messages tailored to the customer can be displayed in real time through the staff's smart glasses, enabling smooth customer interaction.

[1448] "User input information" refers to information about work or tasks that a user provides to the system.

[1449] An "action plan" is a plan that instructs the user on a series of tasks or procedures to be performed based on the input information.

[1450] "Notification" is a message that notifies the user of the generated action plan at an appropriate time.

[1451] A "terminal" is a device that receives information from the system, such as a smartphone, smart glasses, a head-mounted display, or a computer.

[1452] "Means to support the treatment of customers with developmental disabilities" refers to a system that provides support functions to enable staff to provide appropriate assistance to customers with developmental disabilities when they receive services in stores.

[1453] "Means for displaying customer support information in real time" is a function that instantly displays the information necessary when responding to a customer on the staff member's terminal.

[1454] "Profile Information" means information about a user's basic information and characteristics.

[1455] "Means for converting voice input into text data" refers to technology for converting voice input by a user into document data.

[1456] "Means for extracting appropriate actions" is a function that analyzes input data to find appropriate actions or instructions.

[1457] "Reminders" is a feature that notifies users so that they do not forget scheduled actions or events.

[1458] "Means for providing text and voice notifications" refers to a function that notifies users of action instructions and reminders via text message or voice.

[1459] The system required to implement this invention includes the following major components:

[1460] Generating a Program

[1461] The server accepts and analyzes the user's input, generates an action plan, schedules notifications based on the action plan, sends the notifications to the device, and displays information to support interactions with customers with developmental disabilities in real time.

[1462] Program processing explanation

[1463] Hardware and software:

[1464] Hardware: smart glasses, smartphones, computers

[1465] Software: API communication library (requests), speech synthesis library (text_to_speech), generative AI model

[1466] Data processing and calculation flow:

[1467] 1. Acceptance and analysis of user input:

[1468] Users input tasks and schedules through smart glasses or smartphones, using voice input or text input.

[1469] The device receives this input information, converts it to text data if it is voice input, and then analyzes the input data to extract key information (time, action items, etc.).

[1470] 2. Action plan generation and notification schedule:

[1471] The server receives the extracted data and uses a generative AI model to generate an action plan, including the necessary preparations and associated tasks.

[1472] The server schedules the timing of notifications based on the action plan and transmits notification information to the terminal.

[1473] 3. Real-time display of customer support:

[1474] When staff wear smart glasses, the information they need to assist customers is displayed in real time on the glasses' display, and voice notifications are also provided.

[1475] Support messages based on profile information are also provided to help facilitate smooth communication with customers.

[1476] Examples:

[1477] Example 1: Generating an action plan:

[1478] The user speaks, "I have a meeting to explain the new product at 2 p.m."

[1479] The device converts the speech into text and sends it to the server.

[1480] The server analyzes the received text data and generates an action plan (such as sending a reminder five minutes before a meeting or a notification to prepare materials).

[1481] Based on the action plan, the device will notify the employee at 1:55 p.m., "It's five minutes before the meeting. Please prepare your materials," and at 2:00 p.m. will send a reminder, "It's time for the meeting."

[1482] Example 2: Communication support:

[1483] The user enters the text "I have a question about the placement of a new product."

[1484] The terminal analyzes the received text and sends it to the server.

[1485] The server generates a communication support message ("You have a question about the product. What kind of placement would you like?") based on the analyzed data.

[1486] The terminal displays the generated message on the user's smart glasses to support communication with the customer.

[1487] Prompt Sentence Examples

[1488] Sample prompt: "What is the best way to respond to a customer with a developmental disability who asks about new product placement?"

[1489] Example prompt: "Please tell me what you've prepared for the 2pm meeting."

[1490] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1491] Step 1:

[1492] Users input tasks and schedules using smart glasses or a smartphone. Input can be done by voice or text. The device accepts this input and converts it into text data in the case of voice input.

[1493] Input: User voice or text input

[1494] Output: Text data

[1495] Specific operation: The user inputs "I have a meeting to explain the new product at 2 p.m." The device converts the voice into text and generates this text data.

[1496] Step 2:

[1497] The device analyzes the generated text data, extracts key information (task name, time, etc.), and sends it to the server.

[1498] Input: Text data

[1499] Output: Data containing key information

[1500] Specific operation: The terminal analyzes the text "There will be a meeting to explain the new product at 2 p.m.", extracts the key information "2 p.m." and "new product explanation meeting," and sends this to the server.

[1501] Step 3:

[1502] Based on the key information received, the server uses a generative AI model to generate an action plan, including necessary preparations and reminder settings.

[1503] Input: Data containing key information

[1504] Output: Action plan

[1505] Specific operation: Based on the key information about the "meeting to explain the new product at 2 p.m.," the server sets a reminder five minutes before the meeting and generates an action plan to prepare the materials.

[1506] Step 4:

[1507] The server schedules the timing of notifications based on the generated action plan and transmits the notification information to the terminal.

[1508] Input: Action Plan

[1509] Output: Notification schedule and notification information

[1510] Specific operation: Based on the action plan, the server schedules notifications such as "notify preparation of materials at 1:55 pm" and "remind the start of the meeting at 2 pm" and sends that information to the terminal.

[1511] Step 5:

[1512] The device will then notify the user at the specified time based on the notification information received. In the case of smart glasses, notifications are displayed in real time on the screen and, if necessary, are also made via voice.

[1513] Input: Notification information

[1514] Output: Notification to user

[1515] Specific operation: At 1:55 PM, the device displays a message on the smart glasses saying, "It's five minutes until the meeting. Please prepare your materials." It also issues a voice notification. At 2:00 PM, it notifies the user again, saying, "It's time for the meeting."

[1516] Step 6:

[1517] To support the user in communicating with the customer, the server generates a communication support message and transmits it to the terminal, which displays it in real time.

[1518] Input: Customer question or context

[1519] Output: Communication support message

[1520] Specific operation: The user inputs "I have a question about the placement of a new product." Based on this information, the server generates a communication support message saying, "You have a question about the product. What kind of placement would you like?" and sends it to the terminal. The terminal displays this message on the smart glasses in real time, supporting smooth communication with the customer.

[1521] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1522] MODE FOR CARRYING OUT THE INVENTION

[1523] System Overview

[1524] This invention is a support system for solving problems faced by users with developmental disabilities in their daily and social lives. This system accepts and analyzes the user's input information and emotional state, generates an action plan, and presents notifications at appropriate times, providing an environment in which the user can live more smoothly.

[1525] Handling of user profiles

[1526] server

[1527] When a user first launches an application, the server provides a screen for setting user profile information.

[1528] Users enter their personal information (e.g., name, age, characteristics of developmental disabilities, etc.), needs and support methods related to the characteristics, and emotional state.

[1529] Terminal

[1530] The device stores the entered profile information locally and sends it to the server.

[1531] The server stores the received information in a database and generates a profile based on the characteristics.

[1532] Generate an action plan

[1533] User

[1534] The user inputs daily schedules and tasks. This can be done by voice input or text input. For example, "I have a meeting at 1pm."

[1535] Terminal

[1536] The device converts voice input into text data and analyzes it to extract key information, which identifies specific times and action items.

[1537] server

[1538] It receives the analyzed text data and generates an action plan (pre-meeting preparation, reminders, related tasks, etc.) based on the user's characteristics and emotional state.

[1539] The action plan will include specific meeting times, materials to be presented, and necessary preparations.

[1540] Terminal

[1541] The generated action plan is sent to the device and stored locally, where the user can review it and adjust it if necessary.

[1542] Notifications and Reminders

[1543] server

[1544] Based on the generated action plan, schedule the timing of the notification (e.g., 5 minutes before the meeting).

[1545] The notification information is sent to the terminal.

[1546] Terminal

[1547] The reminder will be triggered at the scheduled time, and a notification will be displayed to the user via their device (smartphone, smart glasses, wireless earphones, etc.).

[1548] Communication Support

[1549] User

[1550] When a user needs assistance communicating with others, they can input a specific example (e.g., a conversation with their boss) using voice or text input.

[1551] Terminal

[1552] The device converts the voice into text data and analyzes it.

[1553] server

[1554] Based on the analyzed text data, the system generates an optimal communication support message for the user, such as "Hello, boss. I have a request for you today."

[1555] Terminal

[1556] The generated message is sent to the terminal and provided to the user, who can refer to it for smooth communication.

[1557] Recognizing and responding to emotional states

[1558] User

[1559] While the user is using the system, their emotional state is collected through a facial recognition camera and voice input.

[1560] Terminal

[1561] The collected facial recognition and voice data is analyzed in real time and input into the emotion engine.

[1562] server

[1563] The emotion engine analyzes the input data and identifies the user's current emotional state (happiness, anger, sadness, surprise, etc.).

[1564] The emotion engine generates and adjusts appropriate action plans and communication support messages based on the identified emotional state.

[1565] For example, if a user is feeling stressed, the system will flexibly adjust the timing of reminders and send notifications encouraging them to relax.

[1566] Terminal

[1567] Based on the recognized emotional state, appropriate notifications and support messages are sent to the device and provided to the user in real time.

[1568] Specific examples

[1569] Example 1: Generating an action plan

[1570] A user speaks, "I have a meeting at 1pm."

[1571] The device converts the speech into text and sends it to the server.

[1572] The server analyzes the received text data and generates an action plan.

[1573] The generated action plan includes a five-minute reminder before the meeting and notifications to prepare materials.

[1574] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[1575] Example 2: Communication support

[1576] A user enters the text "I have trouble talking to my boss."

[1577] The terminal analyzes the received text and sends it to the server.

[1578] The server generates a communication support message based on the parsed data.

[1579] The generated message is "Hello boss. I have a favor to ask of you today."

[1580] The terminal displays the generated message to the user to support communication.

[1581] Example 3: Recognizing and responding to emotional states

[1582] Users begin to feel stressed while using the system.

[1583] The device's facial recognition camera and voice input collect emotional states and transmit them to the emotion engine in real time.

[1584] The server's emotion engine recognizes when the user is feeling stressed and generates messages encouraging relaxation, such as "It's time to take a little break."

[1585] The terminal displays the generated message to the user and responds to the user's emotions.

[1586] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics and emotional state of the user.

[1587] The processing flow will be explained below.

[1588] User profile settings

[1589] Step 1:

[1590] User

[1591] Launch the application for the first time and access the profile information input screen.

[1592] Step 2:

[1593] User

[1594] Enter the name, age, characteristics of the developmental disorder (e.g., ADHD, ASD, LD, etc.), and support needs related to the characteristics.

[1595] Step 3:

[1596] Terminal

[1597] The entered profile information is temporarily stored in local storage.

[1598] Step 4:

[1599] Terminal

[1600] Send the information stored in the local storage to the server.

[1601] Step 5:

[1602] server

[1603] The received profile information is stored in a database and a profile based on the characteristics is generated.

[1604] Generate an action plan

[1605] Step 1:

[1606] User

[1607] Enter your daily schedule and tasks (e.g., I have a meeting at 1pm) using voice or text.

[1608] Step 2:

[1609] Terminal

[1610] Converts voice input into text data and analyzes it to extract key information (e.g., date and time, action items).

[1611] Step 3:

[1612] Terminal

[1613] The parsed text data is sent to the server.

[1614] Step 4:

[1615] server

[1616] Based on the received text data, it generates an action plan tailored to each individual's characteristics and needs, including a timeline, preparation list, reminders, and more.

[1617] Step 5:

[1618] server

[1619] The generated action plan is transmitted to the terminal.

[1620] Step 6:

[1621] Terminal

[1622] It stores the received action plan in local storage, notifies the user, and provides the user with the option to adjust the plan if necessary.

[1623] Notifications and Reminders

[1624] Step 1:

[1625] server

[1626] Based on your action plan, schedule notification timing for each action (e.g., a reminder 5 minutes before a meeting).

[1627] Step 2:

[1628] server

[1629] Scheduled notification information is sent to the terminal.

[1630] Step 3:

[1631] Terminal

[1632] Based on the notification timing, the action plan stored in the local storage is checked.

[1633] Step 4:

[1634] Terminal

[1635] A reminder is generated based on the notification timing, and a notification is displayed or sounded to the user via a smartphone, smart glasses, or wireless earphones (for example, at 12:55, a notification is sent saying, "It's five minutes before the meeting. Please prepare your materials.").

[1636] Communication Support

[1637] Step 1:

[1638] User

[1639] If you need help communicating with others (e.g., talking to your boss), enter it by voice or text.

[1640] Step 2:

[1641] Terminal

[1642] Voice input is converted into text data and analyzed to extract specific needs.

[1643] Step 3:

[1644] Terminal

[1645] The extracted needs are transmitted to the server.

[1646] Step 4:

[1647] server

[1648] Based on the received data, the system generates an optimal communication support message for the user. For example, it generates "Hello, boss. I have a request for you today."

[1649] Step 5:

[1650] server

[1651] Send the generated support message to the device.

[1652] Step 6:

[1653] Terminal

[1654] The received message is displayed to the user, and the user is supported to carry out smooth communication based on the message.

[1655] Recognizing and responding to emotional states

[1656] Step 1:

[1657] User

[1658] While using the system, users provide their emotional state through camera and voice input, in real time.

[1659] Step 2:

[1660] Terminal

[1661] The collected facial recognition data and voice data are input into the emotion engine and analysis begins.

[1662] Step 3:

[1663] server

[1664] The emotion engine analyzes facial recognition and voice data to identify the user's emotional state (e.g., joy, anger, sadness, surprise, etc.).

[1665] Step 4:

[1666] server

[1667] Adapt action plans and communication support messages based on the identified emotional state. For example, if the user is feeling stressed, flexibly change the timing of reminders and generate notifications such as "take time to relax."

[1668] Step 5:

[1669] server

[1670] Send coordinated action plans and notifications to your device.

[1671] Step 6:

[1672] Terminal

[1673] Display received notifications and support messages to users and provide appropriate support in real time.

[1674] Specific examples

[1675] Example 1: Generating an action plan

[1676] A user speaks, "I have a meeting at 1pm."

[1677] The device converts the speech into text and sends it to the server.

[1678] The server analyzes the received text data and generates an action plan.

[1679] The generated action plan includes a five-minute reminder before the meeting and notifications to prepare materials.

[1680] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[1681] Example 2: Communication support

[1682] A user enters the text "I have trouble talking to my boss."

[1683] The terminal analyzes the received text and sends it to the server.

[1684] The server generates a communication support message based on the parsed data.

[1685] The generated message is "Hello boss. I have a favor to ask of you today."

[1686] The terminal displays the generated message to the user to support communication.

[1687] Example 3: Recognizing and responding to emotional states

[1688] Users begin to feel stressed while using the system.

[1689] The device's facial recognition camera and voice input collect emotional states and transmit them to the emotion engine in real time.

[1690] The server's emotion engine recognizes when the user is feeling stressed and generates messages encouraging relaxation, such as "It's time to take a little break."

[1691] The terminal displays the generated message to the user and responds to the user's emotions.

[1692] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics and emotional state of the user.

[1693] Example 2

[1694] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1695] There is a need for systems that can efficiently resolve the challenges faced by users with developmental disabilities in their daily and social lives and support them in leading smoother lives. In particular, it is necessary to recognize fluctuations in emotional states and communication difficulties in real time and provide appropriate action plans and support. However, conventional systems have had difficulty fully meeting these requirements and have not been able to provide flexible and appropriate responses that meet the needs of users.

[1696] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1697] In this invention, the server includes means for accepting input information from a user, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, means for sending the notifications to a terminal, means for collecting emotional states through a face recognition camera or voice input, and means for analyzing the emotional states and taking appropriate measures, thereby enabling real-time behavioral and communication support according to the user's characteristics and current emotional state.

[1698] "Means for accepting user input information" refers to a function that provides an interface that allows users to input information into the system in voice, text, or other formats.

[1699] "Means for analyzing input information and generating action plans" refers to algorithms or processes that analyze the input information received from the user and automatically create specific action plans or tasks based on that information.

[1700] The "means for scheduling notifications based on an action plan" is a function for setting notifications at appropriate times according to the contents of the generated action plan.

[1701] "Means for sending notifications to devices" refers to the functionality that delivers scheduled notifications to the user's device (smartphone, tablet, etc.).

[1702] "Means of collecting emotional state through facial recognition cameras and voice input" refers to a system that acquires the user's facial expressions and voice data in real time and determines their emotions based on that.

[1703] "Means for analyzing emotional states and taking appropriate action" refers to the process of analyzing the acquired emotional data and generating an action plan or support message that matches the user's current mood and emotions.

[1704] "Means for setting user profile information" refers to a function that provides an interface that allows users to register and set their own basic information, characteristics, and needs.

[1705] "Means for storing profile information" refers to a function that securely stores the configured user information in a database or local storage.

[1706] "Means for providing support tailored to the user's characteristics" refers to the process of providing assistance and support tailored to individual needs and characteristics based on stored user information.

[1707] "Means for converting voice input into text data" refers to voice recognition technology that converts information input by a user into text data.

[1708] "Means for analyzing text data and extracting appropriate actions" refers to a text analysis algorithm for finding necessary actions and information from voice and text data.

[1709] The "means for generating a reminder based on an action" is a function for creating a reminder (e.g., a schedule notification) according to the extracted action.

[1710] "Means for notifying reminders" refers to a function for displaying set reminders on the user's device.

[1711] "Means for generating and providing messages that support communication with others" refers to the function by which the system automatically generates and provides appropriate sentences and messages to enable users to communicate smoothly.

[1712] This invention is a support system for users with developmental disabilities to solve problems in their daily and social lives. This system accepts input information and emotional state of the user, analyzes it, generates an action plan, and notifies the user at an appropriate time, providing an environment in which the user can live smoothly.

[1713] System Configuration

[1714] This system consists of three elements: a server, a terminal, and a user.

[1715] server

[1716] It provides a profile information setting screen and receives user input information.

[1717] The received information is stored in a database.

[1718] It generates an action plan based on the user's characteristics and emotional state and schedules appropriate notification timing.

[1719] Use an emotion engine to analyze emotional states in real time and take appropriate action.

[1720] Terminal

[1721] The information entered by the user is stored locally and sent to the server.

[1722] It converts voice input into text data and analyzes that data to extract key information.

[1723] Displaying action plans and notifications from the server to the user.

[1724] Emotional states are collected through facial recognition cameras and voice input, analyzed in real time, and sent to the emotion engine.

[1725] User

[1726] Launch the application and enter your profile information.

[1727] Enter daily schedules and tasks using voice or text.

[1728] Enter specific examples of communication with others.

[1729] Review and adjust support messages and action plans from the system, if necessary.

[1730] Hardware and Software

[1731] The specific hardware and software used to realize this system are as follows:

[1732] Hardware

[1733] Server: A central server with a high-performance database and analysis engine.

[1734] Devices: smartphones, tablets, smart glasses, wireless earphones, etc.

[1735] Facial recognition camera: A camera installed on the device.

[1736] Audio input device: The device's built-in microphone.

[1737] software

[1738] Profile settings screen: A web application or mobile application provided on the server side.

[1739] Speech Recognition System: Software for converting voice input into text data.

[1740] Text analysis engine: An AI model for analyzing text data and extracting key information.

[1741] Emotion analysis engine: An AI model that analyzes voice data and facial recognition data to identify emotional states.

[1742] Notification system: Software for scheduling and sending notifications to devices based on action plans.

[1743] Specific examples

[1744] Example 1: Generating an action plan

[1745] 1. The user says, "I have a meeting at 1:00 PM."

[1746] 2. The device converts the speech into text and sends it to the server.

[1747] 3. The server analyzes the received text data and generates an action plan, which includes a five-minute reminder before the meeting and notifications to prepare materials.

[1748] 4. At 12:55, the device will notify you, "It's 5 minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[1749] Example 2: Communication support

[1750] 1. The user types in the text "I have trouble talking to my boss."

[1751] 2. The device analyzes the received text and sends it to the server.

[1752] 3. The server generates a communication support message based on the analyzed data. The generated message is "Hello, boss. I have a request for you today."

[1753] 4. The terminal displays the generated message to the user to support communication.

[1754] Example 3: Recognizing and responding to emotional states

[1755] 1. Users begin to feel stressed while using the system.

[1756] 2. The device's facial recognition camera and voice input collect emotional states and transmit them to the emotion engine in real time.

[1757] 3. The server's emotion engine recognizes that the user is feeling stressed and generates a message encouraging relaxation, such as "It's time to take a short break."

[1758] 4. The device displays the generated message to the user and responds to the user's emotions.

[1759] Prompt Sentence Examples

[1760] "Please explain how users use this system to solve daily and social challenges related to developmental disabilities."

[1761] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics and emotional state of the user.

[1762] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1763] Step 1: Setting up your user profile

[1764] The server provides a user profile information setting screen when the application is first launched. The user enters personal information such as name, age, and characteristics of developmental disabilities. The device stores the entered information locally and sends it to the server. The server stores the received information in a database and generates a profile based on the user's characteristics.

[1765] Input: Personal information (name, age, characteristics of developmental disorders, etc.)

[1766] Data processing: saving and analyzing profile information

[1767] Output: Generate profile and save it to database

[1768] Step 2: Enter your daily schedule and tasks

[1769] Users input their daily schedules and tasks using voice or text. For example, they might input "I have a meeting at 1:00 PM." The device converts the voice input into text data, analyzes the data, extracts specific information (times and action items), and sends it to the server.

[1770] Input: Appointments and tasks (voice or text)

[1771] Data conversion: speech-to-text conversion, text data analysis

[1772] Output: Analysis results, specific times and action items

[1773] Step 3: Generate an action plan

[1774] The server generates an action plan based on the received text data, taking into account the user's characteristics and emotional state. The action plan includes meeting reminders and instructions for preparing materials. The action plan is then sent from the server to the device.

[1775] Input: Parsed text data, user characteristics information

[1776] Data Computation: Generating Action Plans

[1777] Output: Action plan

[1778] Step 4: Communicate and schedule your action plan

[1779] The server schedules the timing of notifications based on the generated action plan. The schedule information is sent to the device, and the device sets the time when the reminder will be sent. For example, at 12:55, the server sends a notification saying, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it sends a reminder saying, "It's time for the meeting."

[1780] Input: Action Plan

[1781] Data Processing: Notification Schedule Settings

[1782] Output: Schedule information, reminder notifications

[1783] Step 5: Support for communicating with others

[1784] When a user needs specific support regarding communication with others, they input a case (e.g., a conversation with their boss). The device converts the received voice into text and sends it to the server. The server analyzes the data, generates a communication support message, and sends it to the device. The device then displays the generated message to the user.

[1785] Input: Communication instance (audio or text)

[1786] Data conversion: speech-to-text conversion, text data analysis

[1787] Output: Support message

[1788] Step 6: Recognize and respond to emotional states

[1789] The system collects the user's emotional state while using the system. The device's facial recognition camera and voice input device send data to the emotion engine in real time. The server's emotion engine analyzes the input data and identifies the user's emotional state. If the server recognizes a specific emotion, such as stress, it generates an appropriate response message and sends it to the device. For example, it displays a message encouraging relaxation, such as "It's time to take a short break."

[1790] Input: Face recognition data, voice input data

[1791] Data analysis: Emotional state analysis

[1792] Output: Response message, adjustment of action plan

[1793] (Application example 2)

[1794] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1795] This invention relates to a support system for solving problems faced by users with developmental disabilities in their daily and social lives. Current technologies are inadequate in providing timely reminders and support messages to users who have difficulties shopping in physical stores or using services. Furthermore, there is a lack of technology that recognizes emotional states and provides appropriate support based on those states in real time. As a result, users are not adequately supported in their daily and social lives.

[1796] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1797] In this invention, the server includes means for accepting user input information, means for analyzing the input information and generating an action plan, means for recognizing the user's emotional state, means for generating reminders and support messages based on the emotional state, and means for generating messages to support communication with others. This allows users with developmental disabilities to receive appropriate reminders and support messages in a timely manner when shopping or using services at physical stores. In addition, appropriate support based on the user's emotional state is provided in real time, and communication support is also provided, thereby reducing the user's stress and difficulties and enabling them to live a smooth daily and social life.

[1798] "User" refers to an individual with a developmental disability who uses the System.

[1799] "Input information" refers to information that a user provides to the system through voice input, text input, or the like.

[1800] An "action plan" is a system-generated plan of specific actions and reminders based on the specific tasks or schedules a user faces.

[1801] "Notifications" are alerts or messages sent to the device at times or in situations specified by the system based on the action plan.

[1802] "Terminal" means a device for receiving notifications from the system, including a smartphone, smart glasses, a head-mounted display, and other electronic devices.

[1803] "Profile Information" means information about you, including personal information such as your name, age, and developmental disability characteristics.

[1804] "Support" refers to assistance including action plans, reminders, support messages, and communication support messages provided according to the user's characteristics and emotional state.

[1805] "Emotional state" refers to the emotions (e.g., joy, anger, sadness, surprise, stress, etc.) that users express while using the system.

[1806] "Emotion recognition" is a technology that identifies a user's emotional state in real time based on data collected from their voice input and facial recognition cameras.

[1807] A "communication support message" is a specific guide message generated by the system to help users communicate smoothly with others.

[1808] A "physical store" is a commercial facility or service provider that has a physical location, where customers visit to purchase goods or use services.

[1809] The present invention describes a specific embodiment of a support system for enabling users with developmental disabilities to smoothly shop and use services in brick-and-mortar stores. This system includes a server and a terminal, and is configured as follows.

[1810] System Overview

[1811] server

[1812] The server includes means for accepting input information from a user, means for analyzing the input information and generating an action plan, means for recognizing the emotional state of the user, means for generating reminders and support messages based on the emotional state, and means for generating messages to support communication with others.

[1813] Terminal

[1814] The devices include smartphones, smart glasses, head-mounted displays, and other electronic devices that receive notifications from the server and display them on their screens. The devices also use cameras and microphones to collect the user's emotional state and send it to the server. Specific software used includes EmotionRecognizer (an emotion recognition engine) and TextToSpeech (a voice synthesis engine).

[1815] Operational Overview

[1816] Profile setup and support

[1817] When a user launches the application for the first time, the server provides a user profile setting screen where the user enters personal information such as name, age, and characteristics, which the device stores locally and then sends to the server.

[1818] The server stores the received information in a database and generates a profile based on the user's characteristics. Based on this profile, the server provides support tailored to the user's characteristics.

[1819] Action plan generation and notification

[1820] When a user inputs their plans for the store (for example, purchasing products or checking in at a service counter) by voice or text, the terminal converts this into text data and sends it to the server.

[1821] The server generates an action plan and related reminders based on the analyzed text data and sends them to the device.

[1822] The device will display a reminder at the specified time, prompting the user to take action.

[1823] Recognizing and responding to emotional states

[1824] The device uses a camera and microphone to collect the user's facial expressions and voice, and inputs this into an emotion recognition engine in real time.

[1825] The server's emotion recognition engine analyzes the user's current emotional state and generates appropriate reminder and support messages based on this. For example, if the user is feeling stressed, it will display a message such as "It's time to take a short break."

[1826] Communication Support

[1827] When a user needs to communicate with others, the user inputs a specific example (for example, the content of a conversation with a store clerk) by voice or text.

[1828] The device analyzes the text data and sends it to the server. The server generates an optimal communication support message based on the analysis results and sends it to the device.

[1829] The terminal displays the generated message to the user, thereby supporting smooth communication.

[1830] Examples and Prompts

[1831] Usage example

[1832] For example, if a user plans to "purchase a product at 1:00 p.m.", the following steps are taken:

[1833] The user speaks, "I'll buy the product at 1pm."

[1834] The terminal converts this into text and sends it to the server.

[1835] The server generates an action plan and generates reminders such as "Start preparing to buy the product at 12:55."

[1836] The device will display a reminder at 12:55, prompting the user to take action.

[1837] If you feel stressed while making a purchase, the device will recognize your emotions in real time and tell you, "It's time to take a break."

[1838] Prompt Sentence Examples

[1839] "Design an application to help customers with developmental disabilities easily shop in brick-and-mortar stores. Use the smartphone's camera and microphone to recognize their emotional state in real time and provide timely reminders and supportive messages."

[1840] As described above, a support system that enables users with developmental disabilities to smoothly shop and use services in physical stores can be realized using a configuration that uses a server and terminals.

[1841] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1842] Step 1:

[1843] When a user launches the application for the first time, they enter their profile settings screen. They enter their personal information, such as their name, age, and characteristics. The device stores this locally and sends it to the server. The entered information is then stored in a database as personal information data.

[1844] Step 2:

[1845] The server analyzes the received user profile information and generates a profile to provide support based on the user's characteristics. The profile includes the user's characteristics and the support items they need, and is then sent to the device. Database search and analysis algorithms are used to generate the profile.

[1846] Step 3:

[1847] The user inputs their plans for the physical store (e.g., "Purchase the product at 1:00 PM") into the system using voice or text. The device converts the voice into text data and sends it to the server. The converted text data is saved as input plan data.

[1848] Step 4:

[1849] The server analyzes the received text data and generates an action plan and reminders using natural language processing and scheduling algorithms. The action plan, which includes specific tasks and their timing, is then sent to the device.

[1850] Step 5:

[1851] The server sets a reminder notification schedule based on the generated action plan. The schedule setting includes adding a timestamp and saving it in an event list. The set schedule is then sent to the device as notification data.

[1852] Step 6:

[1853] The device displays a reminder at a specified time to prompt the user to take action. For example, it displays a message saying, "Please start preparing to purchase the product at 12:55." This can be displayed as a pop-up notification or a sound notification. This triggers movement or action in the physical store.

[1854] Step 7:

[1855] The device uses a camera and microphone to collect the user's emotional state and inputs it into the EmotionRecognizer (emotion recognition engine) in real time. The collected data is saved as facial expression data and voice data.

[1856] Step 8:

[1857] The server uses an emotion recognition engine to analyze the collected facial expression and voice data to identify the user's current emotional state. This analysis uses machine learning algorithms, and the identification result is stored on the server as emotional state data.

[1858] Step 9:

[1859] The server generates appropriate reminders and support messages based on the identified emotional state. For example, if the user is feeling stressed, it generates a message such as "It's time to take a short break." The generated message is then sent to the device as support data.

[1860] Step 10:

[1861] When a user needs support in communicating with others, they input a specific example by voice or text. The device converts this into text data and sends it to the server. The converted text data is saved as communication input data.

[1862] Step 11:

[1863] The server analyzes the received communication input data and generates an optimal communication support message using natural language processing and generative AI models. The generated message is then sent to the device as communication support data.

[1864] Step 12:

[1865] The terminal displays the generated communication support message to the user to support smooth communication. For example, it displays "Hello, store clerk. I need your help." This display completes the support for the user to communicate appropriately.

[1866] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1867] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1868] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1869] [Fourth embodiment]

[1870] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1871] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1872] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1873] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1874] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1875] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1876] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1877] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1878] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1879] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1880] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1881] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1882] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1883] MODE FOR CARRYING OUT THE INVENTION

[1884] System Overview

[1885] This invention is a support system for solving problems faced by users with developmental disabilities in their daily and social lives. This system accepts and analyzes user input, generates an action plan, and presents notifications at appropriate times, providing an environment in which users can live more smoothly.

[1886] Handling of user profiles

[1887] server

[1888] When a user first launches an application, the server provides a screen for setting user profile information.

[1889] Users enter their personal information (e.g., name, age, characteristics of developmental disabilities, etc.), needs related to those characteristics, and support methods.

[1890] Terminal

[1891] The device stores the entered profile information locally and sends it to the server.

[1892] The server stores the received information in a database and generates a profile based on individual characteristics.

[1893] Generate an action plan

[1894] User

[1895] The user inputs daily schedules and tasks. This can be done by voice input or text input. For example, "I have a meeting at 1pm."

[1896] Terminal

[1897] The device converts voice input into text data and analyzes it to extract key information, which identifies specific times and action items.

[1898] server

[1899] It receives the analyzed text data and generates an action plan (pre-meeting preparation, reminders, related tasks, etc.) based on the user's characteristics and needs.

[1900] The action plan will include specific meeting times, materials to be presented, and necessary preparations.

[1901] Terminal

[1902] The generated action plan is sent to the device and stored locally, where the user can review it and adjust it if necessary.

[1903] Notifications and Reminders

[1904] server

[1905] Based on the generated action plan, schedule the timing of the notification (e.g., 5 minutes before the meeting).

[1906] The notification information is sent to the terminal.

[1907] Terminal

[1908] The reminder will be triggered at the scheduled time, and a notification will be displayed to the user via their device (smartphone, smart glasses, wireless earphones, etc.).

[1909] Communication Support

[1910] User

[1911] When a user needs assistance communicating with others, they can input a specific example (e.g., a conversation with their boss) using voice or text input.

[1912] Terminal

[1913] The device converts the voice into text data and analyzes it.

[1914] server

[1915] Based on the analyzed text data, the system generates an optimal communication support message for the user, such as "Hello, boss. I have a request for you today."

[1916] Terminal

[1917] The generated message is sent to the terminal and provided to the user, who can refer to it for smooth communication.

[1918] Specific examples

[1919] Example 1: Generating an action plan

[1920] A user speaks, "I have a meeting at 1pm."

[1921] The device converts the speech into text and sends it to the server.

[1922] The server analyzes the received text data and generates an action plan (such as sending a reminder five minutes before a meeting or a notification to prepare materials).

[1923] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[1924] Example 2: Communication support

[1925] A user enters the text "I have trouble talking to my boss."

[1926] The terminal analyzes the received text and sends it to the server.

[1927] The server generates a communication support message ("Hello, boss. I have a request for you today.") based on the analyzed data.

[1928] The terminal displays the generated message to the user to support communication.

[1929] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics of the user.

[1930] The processing flow will be explained below.

[1931] User profile settings

[1932] Step 1:

[1933] User The user launches the application for the first time and accesses the profile information input screen.

[1934] Step 2:

[1935] User Enter name, age, characteristics of developmental disorder (e.g., ADHD, ASD, LD, etc.), and support needs related to the characteristics.

[1936] Step 3:

[1937] Terminal The entered information is temporarily saved in the terminal's local storage.

[1938] Step 4:

[1939] The information stored in the device's local storage is sent to the server.

[1940] Step 5:

[1941] Server: Stores the received profile information in a database and generates a profile based on the characteristics.

[1942] Generate an action plan

[1943] Step 1:

[1944] User: Enter daily schedules and tasks (e.g., I have a meeting at 1:00 p.m.) using voice or text.

[1945] Step 2:

[1946] Device Converts voice input into text data and analyzes it to extract key information (e.g., date and time, action items).

[1947] Step 3:

[1948] The terminal sends the analyzed text data to the server.

[1949] Step 4:

[1950] Server: Based on the received text data, it generates an action plan tailored to each individual's characteristics and needs, including a timeline, preparation list, reminders, etc.

[1951] Step 5:

[1952] Server: Sends the generated action plan to the terminal.

[1953] Step 6:

[1954] The device stores the received action plan in local storage, notifies the user, and provides the user with the option to adjust the plan if necessary.

[1955] Notifications and Reminders

[1956] Step 1:

[1957] Server Based on the action plan, schedule the notification timing for each action (e.g., a reminder 5 minutes before a meeting).

[1958] Step 2:

[1959] Server: Sends scheduled notification information to the terminal.

[1960] Step 3:

[1961] Device Check the action plan stored in local storage based on the notification timing.

[1962] Step 4:

[1963] Device: A reminder is generated based on the notification timing, and a notification is displayed or audible to the user via a smartphone, smart glasses, or wireless earphones (for example, at 12:55, a notification is sent saying, "It's five minutes before the meeting. Please prepare your materials.").

[1964] Communication Support

[1965] Step 1:

[1966] User When there is something that requires support in communicating with others (e.g., a conversation with a boss), the user inputs it by voice or text.

[1967] Step 2:

[1968] The device converts voice input into text data, analyzes it, and extracts specific needs.

[1969] Step 3:

[1970] The terminal transmits the extracted needs to the server.

[1971] Step 4:

[1972] Server: Based on the received data, it generates the most appropriate communication support message for the user (e.g., "Hello, boss. I have a request for you today.").

[1973] Step 5:

[1974] The server sends the generated support message to the terminal.

[1975] Step 6:

[1976] Terminal: Displays received messages to the user, helping the user to communicate smoothly based on the messages.

[1977] Example 1

[1978] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1979] Users with developmental disabilities face challenges in their daily and social lives, including schedule management and smooth communication. These challenges can cause difficulties in daily and social activities if users do not receive appropriate support. The present invention aims to provide an effective solution to these challenges.

[1980] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1981] In this invention, the server includes means for accepting user input information, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, means for locally saving the generated action plan, and means for displaying notifications to the user. This enables an action plan to be automatically generated based on information input by the user and notifications to be sent at appropriate times. The server also includes means for setting and saving user profile information and means for providing support tailored to the user's characteristics. This enables detailed support tailored to the characteristics of each individual user. The server also includes means for converting voice input into text data, means for analyzing the text data and extracting appropriate actions, and means for generating a communication support message optimized for the user. This also enables smooth communication support through voice input.

[1982] "User" refers to an individual with a developmental disability who uses the system.

[1983] "Input information" refers to data that a user provides to the system, and includes, for example, schedules, tasks, profile information, and the like.

[1984] "Analysis" refers to the processing of data based on input information to understand the content and generate relevant action plans and notifications.

[1985] An "action plan" is a list of schedules and tasks generated based on input information, and is intended to support users' daily and social lives.

[1986] "Notification" means information provided to a user at an appropriate time based on an action plan, including reminders and alerts.

[1987] "Schedule" refers to the time and order in which notifications are set based on an action plan.

[1988] A "terminal" is a hardware device for running the system, including smartphones and tablets.

[1989] "Local storage" refers to the temporary or long-term storage of data within a device.

[1990] "Profile information" refers to personal information provided by the user to the system, including name, age, and characteristics of the developmental disorder.

[1991] "Support" refers to providing assistance tailored to the characteristics and needs of users.

[1992] "Voice input" refers to the means by which a user provides information to a system using speech.

[1993] "Text data" refers to data obtained by converting voice input into text information.

[1994] "Communication support messages" are messages generated based on analyzed data to help users communicate smoothly with others.

[1995] This invention is a system for solving problems faced by users with developmental disabilities in their daily and social lives. This system accepts and analyzes information input by the user, generates an action plan, and presents notifications at appropriate times, thereby providing an environment in which the user can live more smoothly.

[1996] Configuration and Operation Procedures

[1997] 1. Handling of User Profiles

[1998] When a user first launches the application, the server provides a user profile information setup screen, where the user enters their name, age, characteristics of their developmental disorder, support needs, etc. The device then stores the entered profile information locally and sends it to the server. The server then stores the received information in a database and generates a profile based on each individual's characteristics.

[1999] 2. Generate an action plan

[2000] The user inputs daily schedules and tasks using voice or text. For example, "I have a meeting at 1 p.m." The device converts the voice input into text data and analyzes it to extract key information. This analysis identifies specific times and action items. The server receives the analyzed text data and generates an action plan (pre-meeting preparations, reminders, and related tasks) based on the user's characteristics and needs. The generated action plan may include, for example, "send a reminder 5 minutes before the meeting and a notification to prepare materials." The action plan is then sent to the device and stored locally. The user can review it and adjust it as needed.

[2001] 3. Notifications and Reminders

[2002] The server schedules the notification based on the generated action plan. For example, it sets a reminder five minutes before a meeting. The notification information is sent to the device. The device triggers the reminder at the scheduled time, and a notification is displayed to the user via their smartphone or other device.

[2003] 4. Communication support

[2004] When a user needs support in communicating with others, they input a specific example (e.g., the contents of a conversation with their boss). This is done by voice input or text input. The device converts the voice into text data and analyzes it. The server generates an optimal communication support message for the user based on the analyzed text data. For example, it generates a specific message such as, "Hello, boss. I have a request for you today." The device displays the generated message to the user to support communication.

[2005] Specific examples

[2006] Example 1: Generating an action plan

[2007] A user speaks, "I have a meeting at 1pm."

[2008] The device converts the speech into text and sends it to the server.

[2009] The server analyzes the received text data and generates an action plan (such as sending a reminder five minutes before a meeting or a notification to prepare materials).

[2010] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[2011] Example 2: Communication support

[2012] A user enters the text "I have trouble talking to my boss."

[2013] The terminal analyzes the received text and sends it to the server.

[2014] The server generates a communication support message ("Hello, boss. I have a request for you today.") based on the analyzed data.

[2015] The terminal displays the generated message to the user to support communication.

[2016] Prompt Sentence Examples

[2017] "Please explain in natural language the process of dictating that there will be a meeting at 1pm."

[2018] "Please explain in detail the process for generating support messages for users who are uncomfortable speaking with their managers."

[2019] Through these steps, the system of the present invention provides an environment in which users with developmental disabilities can overcome challenges in their daily and social lives and live more smoothly.

[2020] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2021] Step 1:

[2022] The user initially launches the application, which starts the system operation and provides an input that displays a profile setting screen for the user.

[2023] Step 2:

[2024] The server provides a user profile information setup screen, allowing the user to input their name, age, characteristics of their developmental disorder, support needs, etc. This input leads to an output that captures the user's basic personal information.

[2025] Step 3:

[2026] The user enters personal information such as name, age, and characteristics of the developmental disorder. This input provides the system with basic data to provide support tailored to the user's characteristics. This data is then used in later steps to generate an action plan and support messages tailored to the user's characteristics.

[2027] Step 4:

[2028] The device locally stores the entered profile information and transmits it to the server. The input is user information, and the local storage and transmission process communicates this to the server as output.

[2029] Step 5:

[2030] The server stores the received information in a database and generates a profile based on individual characteristics. In this process, the entered user information is received and stored in a database. At this time, a profile is generated based on the stored data and stored in the server.

[2031] Step 6:

[2032] The user inputs daily schedules and tasks by voice or text. The input here is specific schedule and task information, such as "I have a meeting at 1:00 p.m." The output is generated as text data.

[2033] Step 7:

[2034] The device converts voice input into text data and sends that data to the server. Voice data is input, and text data is output through voice recognition. This text data is then sent to the server.

[2035] Step 8:

[2036] The server analyzes the received text data and generates an action plan. This analysis takes the text data as input, extracts key information using natural language processing, and outputs a notification plan. The generated action plan includes specific content, such as "send a reminder five minutes before the meeting and a notification to prepare materials."

[2037] Step 9:

[2038] The server sends the generated action plan to the terminal. The input here is the generated action plan, and the output is notification data sent to the terminal.

[2039] Step 10:

[2040] The terminal stores the action plan locally and displays it to the user, with the received action plan as input and the action plan stored locally and displayed on the user interface as output.

[2041] Step 11:

[2042] The server schedules notifications based on the generated action plan, where the input is the content of the action plan and the output is the schedule for each notification.

[2043] Step 12:

[2044] The server sends scheduled notification information to the terminal, which has the schedule information as input and sends it to the terminal as output.

[2045] Step 13:

[2046] The device triggers a reminder based on the schedule and displays a notification to the user. The input here is schedule information, and the output is a reminder that is triggered and notified to the user in real time.

[2047] Step 14:

[2048] When a user needs support in communicating with others, they input a specific example. For example, they input "the contents of a conversation with their boss." The input can be a conversation scenario, and text data is generated as the output.

[2049] Step 15:

[2050] The device converts the voice into text data and sends it to the server. The input is voice data, and the output is text data, which is then sent to the server.

[2051] Step 16:

[2052] The server analyzes the received text data and generates the optimal communication support message. The input is text data, and an appropriate message is output through natural language processing. For example, "Hello, boss. I have a request for you today."

[2053] Step 17:

[2054] The server generates and sends messages to the terminal. It has the generated message as input and sends it to the terminal as output.

[2055] Step 18:

[2056] The terminal displays the message to the user. The received message is the input, and the output is displayed on the user interface. The user can use this as a reference for communication.

[2057] (Application example 1)

[2058] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2059] There is a need to solve the problems that users with developmental disabilities face in their daily and social lives, and to realize smooth communication and efficient support between staff and customers with developmental disabilities, especially in brick-and-mortar stores. However, it is difficult for staff to instantly determine the appropriate response method based on the characteristics of each customer, which leads to a decrease in customer satisfaction and an increase in the burden on staff.

[2060] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2061] In this invention, the server includes means for accepting user input information, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, means for sending the notifications to a terminal, means for supporting interactions with customers with developmental disabilities, and means for displaying the customer interaction support information in real time. This enables store staff to instantly determine appropriate responses for customers with developmental disabilities and provide efficient support. Specifically, action plans and communication support messages tailored to the customer can be displayed in real time through the staff's smart glasses, enabling smooth customer interaction.

[2062] "User input information" refers to information about work or tasks that a user provides to the system.

[2063] An "action plan" is a plan that instructs the user on a series of tasks or procedures to be performed based on the input information.

[2064] "Notification" is a message that notifies the user of the generated action plan at an appropriate time.

[2065] A "terminal" is a device that receives information from the system, such as a smartphone, smart glasses, a head-mounted display, or a computer.

[2066] "Means to support the treatment of customers with developmental disabilities" refers to a system that provides support functions to enable staff to provide appropriate assistance to customers with developmental disabilities when they receive services in stores.

[2067] "Means for displaying customer support information in real time" is a function that instantly displays the information necessary when responding to a customer on the staff member's terminal.

[2068] "Profile Information" means information about a user's basic information and characteristics.

[2069] "Means for converting voice input into text data" refers to technology for converting voice input by a user into document data.

[2070] "Means for extracting appropriate actions" is a function that analyzes input data to find appropriate actions or instructions.

[2071] "Reminders" is a feature that notifies users so that they do not forget scheduled actions or events.

[2072] "Means for providing text and voice notifications" refers to a function that notifies users of action instructions and reminders via text message or voice.

[2073] The system required to implement this invention includes the following major components:

[2074] Generating a Program

[2075] The server accepts and analyzes the user's input, generates an action plan, schedules notifications based on the action plan, sends the notifications to the device, and displays information to support interactions with customers with developmental disabilities in real time.

[2076] Program processing explanation

[2077] Hardware and software:

[2078] Hardware: smart glasses, smartphones, computers

[2079] Software: API communication library (requests), speech synthesis library (text_to_speech), generative AI model

[2080] Data processing and calculation flow:

[2081] 1. Acceptance and analysis of user input:

[2082] Users input tasks and schedules through smart glasses or smartphones, using voice input or text input.

[2083] The device receives this input information, converts it to text data if it is voice input, and then analyzes the input data to extract key information (time, action items, etc.).

[2084] 2. Action plan generation and notification schedule:

[2085] The server receives the extracted data and uses a generative AI model to generate an action plan, including the necessary preparations and associated tasks.

[2086] The server schedules the timing of notifications based on the action plan and transmits notification information to the terminal.

[2087] 3. Real-time display of customer support:

[2088] When staff wear smart glasses, the information they need to assist customers is displayed in real time on the glasses' display, and voice notifications are also provided.

[2089] Support messages based on profile information are also provided to help facilitate smooth communication with customers.

[2090] Examples:

[2091] Example 1: Generating an action plan:

[2092] The user speaks, "I have a meeting to explain the new product at 2 p.m."

[2093] The device converts the speech into text and sends it to the server.

[2094] The server analyzes the received text data and generates an action plan (such as sending a reminder five minutes before a meeting or a notification to prepare materials).

[2095] Based on the action plan, the device will notify the employee at 1:55 p.m., "It's five minutes before the meeting. Please prepare your materials," and at 2:00 p.m. will send a reminder, "It's time for the meeting."

[2096] Example 2: Communication support:

[2097] The user enters the text "I have a question about the placement of a new product."

[2098] The terminal analyzes the received text and sends it to the server.

[2099] The server generates a communication support message ("You have a question about the product. What kind of placement would you like?") based on the analyzed data.

[2100] The terminal displays the generated message on the user's smart glasses to support communication with the customer.

[2101] Prompt Sentence Examples

[2102] Sample prompt: "What is the best way to respond to a customer with a developmental disability who asks about new product placement?"

[2103] Example prompt: "Please tell me what you've prepared for the 2pm meeting."

[2104] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2105] Step 1:

[2106] Users input tasks and schedules using smart glasses or a smartphone. Input can be done by voice or text. The device accepts this input and converts it into text data in the case of voice input.

[2107] Input: User voice or text input

[2108] Output: Text data

[2109] Specific operation: The user inputs "I have a meeting to explain the new product at 2 p.m." The device converts the voice into text and generates this text data.

[2110] Step 2:

[2111] The device analyzes the generated text data, extracts key information (task name, time, etc.), and sends it to the server.

[2112] Input: Text data

[2113] Output: Data containing key information

[2114] Specific operation: The terminal analyzes the text "There will be a meeting to explain the new product at 2 p.m.", extracts the key information "2 p.m." and "new product explanation meeting," and sends this to the server.

[2115] Step 3:

[2116] Based on the key information received, the server uses a generative AI model to generate an action plan, including necessary preparations and reminder settings.

[2117] Input: Data containing key information

[2118] Output: Action plan

[2119] Specific operation: Based on the key information about the "meeting to explain the new product at 2 p.m.," the server sets a reminder five minutes before the meeting and generates an action plan to prepare the materials.

[2120] Step 4:

[2121] The server schedules the timing of notifications based on the generated action plan and transmits the notification information to the terminal.

[2122] Input: Action Plan

[2123] Output: Notification schedule and notification information

[2124] Specific operation: Based on the action plan, the server schedules notifications such as "notify preparation of materials at 1:55 pm" and "remind the start of the meeting at 2 pm" and sends that information to the terminal.

[2125] Step 5:

[2126] The device will then notify the user at the specified time based on the notification information received. In the case of smart glasses, notifications are displayed in real time on the screen and, if necessary, are also made via voice.

[2127] Input: Notification information

[2128] Output: Notification to user

[2129] Specific operation: At 1:55 PM, the device displays a message on the smart glasses saying, "It's five minutes until the meeting. Please prepare your materials." It also issues a voice notification. At 2:00 PM, it notifies the user again, saying, "It's time for the meeting."

[2130] Step 6:

[2131] To support the user in communicating with the customer, the server generates a communication support message and transmits it to the terminal, which displays it in real time.

[2132] Input: Customer question or context

[2133] Output: Communication support message

[2134] Specific operation: The user inputs "I have a question about the placement of a new product." Based on this information, the server generates a communication support message saying, "You have a question about the product. What kind of placement would you like?" and sends it to the terminal. The terminal displays this message on the smart glasses in real time, supporting smooth communication with the customer.

[2135] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2136] MODE FOR CARRYING OUT THE INVENTION

[2137] System Overview

[2138] This invention is a support system for solving problems faced by users with developmental disabilities in their daily and social lives. This system accepts and analyzes the user's input information and emotional state, generates an action plan, and presents notifications at appropriate times, providing an environment in which the user can live more smoothly.

[2139] Handling of user profiles

[2140] server

[2141] When a user first launches an application, the server provides a screen for setting user profile information.

[2142] Users enter their personal information (e.g., name, age, characteristics of developmental disabilities, etc.), needs and support methods related to the characteristics, and emotional state.

[2143] Terminal

[2144] The device stores the entered profile information locally and sends it to the server.

[2145] The server stores the received information in a database and generates a profile based on the characteristics.

[2146] Generate an action plan

[2147] User

[2148] The user inputs daily schedules and tasks. This can be done by voice input or text input. For example, "I have a meeting at 1pm."

[2149] Terminal

[2150] The device converts voice input into text data and analyzes it to extract key information, which identifies specific times and action items.

[2151] server

[2152] It receives the analyzed text data and generates an action plan (pre-meeting preparation, reminders, related tasks, etc.) based on the user's characteristics and emotional state.

[2153] The action plan will include specific meeting times, materials to be presented, and necessary preparations.

[2154] Terminal

[2155] The generated action plan is sent to the device and stored locally, where the user can review it and adjust it if necessary.

[2156] Notifications and Reminders

[2157] server

[2158] Based on the generated action plan, schedule the timing of the notification (e.g., 5 minutes before the meeting).

[2159] The notification information is sent to the terminal.

[2160] Terminal

[2161] The reminder will be triggered at the scheduled time, and a notification will be displayed to the user via their device (smartphone, smart glasses, wireless earphones, etc.).

[2162] Communication Support

[2163] User

[2164] When a user needs assistance communicating with others, they can input a specific example (e.g., a conversation with their boss) using voice or text input.

[2165] Terminal

[2166] The device converts the voice into text data and analyzes it.

[2167] server

[2168] Based on the analyzed text data, the system generates an optimal communication support message for the user, such as "Hello, boss. I have a request for you today."

[2169] Terminal

[2170] The generated message is sent to the terminal and provided to the user, who can refer to it for smooth communication.

[2171] Recognizing and responding to emotional states

[2172] User

[2173] While the user is using the system, their emotional state is collected through a facial recognition camera and voice input.

[2174] Terminal

[2175] The collected facial recognition and voice data is analyzed in real time and input into the emotion engine.

[2176] server

[2177] The emotion engine analyzes the input data and identifies the user's current emotional state (happiness, anger, sadness, surprise, etc.).

[2178] The emotion engine generates and adjusts appropriate action plans and communication support messages based on the identified emotional state.

[2179] For example, if a user is feeling stressed, the system will flexibly adjust the timing of reminders and send notifications encouraging them to relax.

[2180] Terminal

[2181] Based on the recognized emotional state, appropriate notifications and support messages are sent to the device and provided to the user in real time.

[2182] Specific examples

[2183] Example 1: Generating an action plan

[2184] A user speaks, "I have a meeting at 1pm."

[2185] The device converts the speech into text and sends it to the server.

[2186] The server analyzes the received text data and generates an action plan.

[2187] The generated action plan includes a five-minute reminder before the meeting and notifications to prepare materials.

[2188] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[2189] Example 2: Communication support

[2190] A user enters the text "I have trouble talking to my boss."

[2191] The terminal analyzes the received text and sends it to the server.

[2192] The server generates a communication support message based on the parsed data.

[2193] The generated message is "Hello boss. I have a favor to ask of you today."

[2194] The terminal displays the generated message to the user to support communication.

[2195] Example 3: Recognizing and responding to emotional states

[2196] Users begin to feel stressed while using the system.

[2197] The device's facial recognition camera and voice input collect emotional states and transmit them to the emotion engine in real time.

[2198] The server's emotion engine recognizes when the user is feeling stressed and generates messages encouraging relaxation, such as "It's time to take a little break."

[2199] The terminal displays the generated message to the user and responds to the user's emotions.

[2200] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics and emotional state of the user.

[2201] The processing flow will be explained below.

[2202] User profile settings

[2203] Step 1:

[2204] User

[2205] Launch the application for the first time and access the profile information input screen.

[2206] Step 2:

[2207] User

[2208] Enter the name, age, characteristics of the developmental disorder (e.g., ADHD, ASD, LD, etc.), and support needs related to the characteristics.

[2209] Step 3:

[2210] Terminal

[2211] The entered profile information is temporarily stored in local storage.

[2212] Step 4:

[2213] Terminal

[2214] Send the information stored in the local storage to the server.

[2215] Step 5:

[2216] server

[2217] The received profile information is stored in a database and a profile based on the characteristics is generated.

[2218] Generate an action plan

[2219] Step 1:

[2220] User

[2221] Enter your daily schedule and tasks (e.g., I have a meeting at 1pm) using voice or text.

[2222] Step 2:

[2223] Terminal

[2224] Converts voice input into text data and analyzes it to extract key information (e.g., date and time, action items).

[2225] Step 3:

[2226] Terminal

[2227] The parsed text data is sent to the server.

[2228] Step 4:

[2229] server

[2230] Based on the received text data, it generates an action plan tailored to each individual's characteristics and needs, including a timeline, preparation list, reminders, and more.

[2231] Step 5:

[2232] server

[2233] The generated action plan is transmitted to the terminal.

[2234] Step 6:

[2235] Terminal

[2236] It stores the received action plan in local storage, notifies the user, and provides the user with the option to adjust the plan if necessary.

[2237] Notifications and Reminders

[2238] Step 1:

[2239] server

[2240] Based on your action plan, schedule notification timing for each action (e.g., a reminder 5 minutes before a meeting).

[2241] Step 2:

[2242] server

[2243] Scheduled notification information is sent to the terminal.

[2244] Step 3:

[2245] Terminal

[2246] Based on the notification timing, the action plan stored in the local storage is checked.

[2247] Step 4:

[2248] Terminal

[2249] A reminder is generated based on the notification timing, and a notification is displayed or sounded to the user via a smartphone, smart glasses, or wireless earphones (for example, at 12:55, a notification is sent saying, "It's five minutes before the meeting. Please prepare your materials.").

[2250] Communication Support

[2251] Step 1:

[2252] User

[2253] If you need help communicating with others (e.g., talking to your boss), enter it by voice or text.

[2254] Step 2:

[2255] Terminal

[2256] Voice input is converted into text data and analyzed to extract specific needs.

[2257] Step 3:

[2258] Terminal

[2259] The extracted needs are transmitted to the server.

[2260] Step 4:

[2261] server

[2262] Based on the received data, the system generates an optimal communication support message for the user. For example, it generates "Hello, boss. I have a request for you today."

[2263] Step 5:

[2264] server

[2265] Send the generated support message to the device.

[2266] Step 6:

[2267] Terminal

[2268] The received message is displayed to the user, and the user is supported to carry out smooth communication based on the message.

[2269] Recognizing and responding to emotional states

[2270] Step 1:

[2271] User

[2272] While using the system, users provide their emotional state through camera and voice input, in real time.

[2273] Step 2:

[2274] Terminal

[2275] The collected facial recognition data and voice data are input into the emotion engine and analysis begins.

[2276] Step 3:

[2277] server

[2278] The emotion engine analyzes facial recognition and voice data to identify the user's emotional state (e.g., joy, anger, sadness, surprise, etc.).

[2279] Step 4:

[2280] server

[2281] Adapt action plans and communication support messages based on the identified emotional state. For example, if the user is feeling stressed, flexibly change the timing of reminders and generate notifications such as "take time to relax."

[2282] Step 5:

[2283] server

[2284] Send coordinated action plans and notifications to your device.

[2285] Step 6:

[2286] Terminal

[2287] Display received notifications and support messages to users and provide appropriate support in real time.

[2288] Specific examples

[2289] Example 1: Generating an action plan

[2290] A user speaks, "I have a meeting at 1pm."

[2291] The device converts the speech into text and sends it to the server.

[2292] The server analyzes the received text data and generates an action plan.

[2293] The generated action plan includes a five-minute reminder before the meeting and notifications to prepare materials.

[2294] At 12:55, the device will notify you, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[2295] Example 2: Communication support

[2296] A user enters the text "I have trouble talking to my boss."

[2297] The terminal analyzes the received text and sends it to the server.

[2298] The server generates a communication support message based on the parsed data.

[2299] The generated message is "Hello boss. I have a favor to ask of you today."

[2300] The terminal displays the generated message to the user to support communication.

[2301] Example 3: Recognizing and responding to emotional states

[2302] Users begin to feel stressed while using the system.

[2303] The device's facial recognition camera and voice input collect emotional states and transmit them to the emotion engine in real time.

[2304] The server's emotion engine recognizes when the user is feeling stressed and generates messages encouraging relaxation, such as "It's time to take a little break."

[2305] The terminal displays the generated message to the user and responds to the user's emotions.

[2306] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics and emotional state of the user.

[2307] Example 2

[2308] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2309] There is a need for systems that can efficiently resolve the challenges faced by users with developmental disabilities in their daily and social lives and support them in leading smoother lives. In particular, it is necessary to recognize fluctuations in emotional states and communication difficulties in real time and provide appropriate action plans and support. However, conventional systems have had difficulty fully meeting these requirements and have not been able to provide flexible and appropriate responses that meet the needs of users.

[2310] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2311] In this invention, the server includes means for accepting input information from a user, means for analyzing the input information and generating an action plan, means for scheduling notifications based on the action plan, means for sending the notifications to a terminal, means for collecting emotional states through a face recognition camera or voice input, and means for analyzing the emotional states and taking appropriate measures, thereby enabling real-time behavioral and communication support according to the user's characteristics and current emotional state.

[2312] "Means for accepting user input information" refers to a function that provides an interface that allows users to input information into the system in voice, text, or other formats.

[2313] "Means for analyzing input information and generating action plans" refers to algorithms or processes that analyze the input information received from the user and automatically create specific action plans or tasks based on that information.

[2314] The "means for scheduling notifications based on an action plan" is a function for setting notifications at appropriate times according to the contents of the generated action plan.

[2315] "Means for sending notifications to devices" refers to the functionality that delivers scheduled notifications to the user's device (smartphone, tablet, etc.).

[2316] "Means of collecting emotional state through facial recognition cameras and voice input" refers to a system that acquires the user's facial expressions and voice data in real time and determines their emotions based on that.

[2317] "Means for analyzing emotional states and taking appropriate action" refers to the process of analyzing the acquired emotional data and generating an action plan or support message that matches the user's current mood and emotions.

[2318] "Means for setting user profile information" refers to a function that provides an interface that allows users to register and set their own basic information, characteristics, and needs.

[2319] "Means for storing profile information" refers to a function that securely stores the configured user information in a database or local storage.

[2320] "Means for providing support tailored to the user's characteristics" refers to the process of providing assistance and support tailored to individual needs and characteristics based on stored user information.

[2321] "Means for converting voice input into text data" refers to voice recognition technology that converts information input by a user into text data.

[2322] "Means for analyzing text data and extracting appropriate actions" refers to a text analysis algorithm for finding necessary actions and information from voice and text data.

[2323] The "means for generating a reminder based on an action" is a function for creating a reminder (e.g., a schedule notification) according to the extracted action.

[2324] "Means for notifying reminders" refers to a function for displaying set reminders on the user's device.

[2325] "Means for generating and providing messages that support communication with others" refers to the function by which the system automatically generates and provides appropriate sentences and messages to enable users to communicate smoothly.

[2326] This invention is a support system for users with developmental disabilities to solve problems in their daily and social lives. This system accepts input information and emotional state of the user, analyzes it, generates an action plan, and notifies the user at an appropriate time, providing an environment in which the user can live smoothly.

[2327] System Configuration

[2328] This system consists of three elements: a server, a terminal, and a user.

[2329] server

[2330] It provides a profile information setting screen and receives user input information.

[2331] The received information is stored in a database.

[2332] It generates an action plan based on the user's characteristics and emotional state and schedules appropriate notification timing.

[2333] Use an emotion engine to analyze emotional states in real time and take appropriate action.

[2334] Terminal

[2335] The information entered by the user is stored locally and sent to the server.

[2336] It converts voice input into text data and analyzes that data to extract key information.

[2337] Displaying action plans and notifications from the server to the user.

[2338] Emotional states are collected through facial recognition cameras and voice input, analyzed in real time, and sent to the emotion engine.

[2339] User

[2340] Launch the application and enter your profile information.

[2341] Enter daily schedules and tasks using voice or text.

[2342] Enter specific examples of communication with others.

[2343] Review and adjust support messages and action plans from the system, if necessary.

[2344] Hardware and Software

[2345] The specific hardware and software used to realize this system are as follows:

[2346] Hardware

[2347] Server: A central server with a high-performance database and analysis engine.

[2348] Devices: smartphones, tablets, smart glasses, wireless earphones, etc.

[2349] Facial recognition camera: A camera installed on the device.

[2350] Audio input device: The device's built-in microphone.

[2351] software

[2352] Profile settings screen: A web application or mobile application provided on the server side.

[2353] Speech Recognition System: Software for converting voice input into text data.

[2354] Text analysis engine: An AI model for analyzing text data and extracting key information.

[2355] Emotion analysis engine: An AI model that analyzes voice data and facial recognition data to identify emotional states.

[2356] Notification system: Software for scheduling and sending notifications to devices based on action plans.

[2357] Specific examples

[2358] Example 1: Generating an action plan

[2359] 1. The user says, "I have a meeting at 1:00 PM."

[2360] 2. The device converts the speech into text and sends it to the server.

[2361] 3. The server analyzes the received text data and generates an action plan, which includes a five-minute reminder before the meeting and notifications to prepare materials.

[2362] 4. At 12:55, the device will notify you, "It's 5 minutes before the meeting. Please prepare your materials," and at 1:00, it will send a reminder, "It's time for the meeting."

[2363] Example 2: Communication support

[2364] 1. The user types in the text "I have trouble talking to my boss."

[2365] 2. The device analyzes the received text and sends it to the server.

[2366] 3. The server generates a communication support message based on the analyzed data. The generated message is "Hello, boss. I have a request for you today."

[2367] 4. The terminal displays the generated message to the user to support communication.

[2368] Example 3: Recognizing and responding to emotional states

[2369] 1. Users begin to feel stressed while using the system.

[2370] 2. The device's facial recognition camera and voice input collect emotional states and transmit them to the emotion engine in real time.

[2371] 3. The server's emotion engine recognizes that the user is feeling stressed and generates a message encouraging relaxation, such as "It's time to take a short break."

[2372] 4. The device displays the generated message to the user and responds to the user's emotions.

[2373] Prompt Sentence Examples

[2374] "Please explain how users use this system to solve daily and social challenges related to developmental disabilities."

[2375] As described above, the present invention provides a system that realizes behavioral support and communication support according to the characteristics and emotional state of the user.

[2376] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2377] Step 1: Setting up your user profile

[2378] The server provides a user profile information setting screen when the application is first launched. The user enters personal information such as name, age, and characteristics of developmental disabilities. The device stores the entered information locally and sends it to the server. The server stores the received information in a database and generates a profile based on the user's characteristics.

[2379] Input: Personal information (name, age, characteristics of developmental disorders, etc.)

[2380] Data processing: saving and analyzing profile information

[2381] Output: Generate profile and save it to database

[2382] Step 2: Enter your daily schedule and tasks

[2383] Users input their daily schedules and tasks using voice or text. For example, they might input "I have a meeting at 1:00 PM." The device converts the voice input into text data, analyzes the data, extracts specific information (times and action items), and sends it to the server.

[2384] Input: Appointments and tasks (voice or text)

[2385] Data conversion: speech-to-text conversion, text data analysis

[2386] Output: Analysis results, specific times and action items

[2387] Step 3: Generate an action plan

[2388] The server generates an action plan based on the received text data, taking into account the user's characteristics and emotional state. The action plan includes meeting reminders and instructions for preparing materials. The action plan is then sent from the server to the device.

[2389] Input: Parsed text data, user characteristics information

[2390] Data Computation: Generating Action Plans

[2391] Output: Action plan

[2392] Step 4: Communicate and schedule your action plan

[2393] The server schedules the timing of notifications based on the generated action plan. The schedule information is sent to the device, and the device sets the time when the reminder will be sent. For example, at 12:55, the server sends a notification saying, "It's five minutes before the meeting. Please prepare your materials," and at 1:00, it sends a reminder saying, "It's time for the meeting."

[2394] Input: Action Plan

[2395] Data Processing: Notification Schedule Settings

[2396] Output: Schedule information, reminder notifications

[2397] Step 5: Support for communicating with others

[2398] When a user needs specific support regarding communication with others, they input a case (e.g., a conversation with their boss). The device converts the received voice into text and sends it to the server. The server analyzes the data, generates a communication support message, and sends it to the device. The device then displays the generated message to the user.

[2399] Input: Communication instance (audio or text)

[2400] Data conversion: speech-to-text conversion, text data analysis

[2401] Output: Support message

[2402] Step 6: Recognize and respond to emotional states

[2403] The system collects the user's emotional state while using the system. The device's facial recognition camera and voice input device send data to the emotion engine in real time. The server's emotion engine analyzes the input data and identifies the user's emotional state. If the server recognizes a specific emotion, such as stress, it generates an appropriate response message and sends it to the device. For example, it displays a message encouraging relaxation, such as "It's time to take a short break."

[2404] Input: Face recognition data, voice input data

[2405] Data analysis: Emotional state analysis

[2406] Output: Response message, adjustment of action plan

[2407] (Application example 2)

[2408] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2409] This invention relates to a support system for solving problems faced by users with developmental disabilities in their daily and social lives. Current technologies are inadequate in providing timely reminders and support messages to users who have difficulties shopping in physical stores or using services. Furthermore, there is a lack of technology that recognizes emotional states and provides appropriate support based on those states in real time. As a result, users are not adequately supported in their daily and social lives.

[2410] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2411] In this invention, the server includes means for accepting user input information, means for analyzing the input information and generating an action plan, means for recognizing the user's emotional state, means for generating reminders and support messages based on the emotional state, and means for generating messages to support communication with others. This allows users with developmental disabilities to receive appropriate reminders and support messages in a timely manner when shopping or using services at physical stores. In addition, appropriate support based on the user's emotional state is provided in real time, and communication support is also provided, thereby reducing the user's stress and difficulties and enabling them to live a smooth daily and social life.

[2412] "User" refers to an individual with a developmental disability who uses the System.

[2413] "Input information" refers to information that a user provides to the system through voice input, text input, or the like.

[2414] An "action plan" is a system-generated plan of specific actions and reminders based on the specific tasks or schedules a user faces.

[2415] "Notifications" are alerts or messages sent to the device at times or in situations specified by the system based on the action plan.

[2416] "Terminal" means a device for receiving notifications from the system, including a smartphone, smart glasses, a head-mounted display, and other electronic devices.

[2417] "Profile Information" means information about you, including personal information such as your name, age, and developmental disability characteristics.

[2418] "Support" refers to assistance including action plans, reminders, support messages, and communication support messages provided according to the user's characteristics and emotional state.

[2419] "Emotional state" refers to the emotions (e.g., joy, anger, sadness, surprise, stress, etc.) that users express while using the system.

[2420] "Emotion recognition" is a technology that identifies a user's emotional state in real time based on data collected from their voice input and facial recognition cameras.

[2421] A "communication support message" is a specific guide message generated by the system to help users communicate smoothly with others.

[2422] A "physical store" is a commercial facility or service provider that has a physical location, where customers visit to purchase goods or use services.

[2423] The present invention describes a specific embodiment of a support system for enabling users with developmental disabilities to smoothly shop and use services in brick-and-mortar stores. This system includes a server and a terminal, and is configured as follows.

[2424] System Overview

[2425] server

[2426] The server includes means for accepting input information from a user, means for analyzing the input information and generating an action plan, means for recognizing the emotional state of the user, means for generating reminders and support messages based on the emotional state, and means for generating messages to support communication with others.

[2427] Terminal

[2428] The devices include smartphones, smart glasses, head-mounted displays, and other electronic devices that receive notifications from the server and display them on their screens. The devices also use cameras and microphones to collect the user's emotional state and send it to the server. Specific software used includes EmotionRecognizer (an emotion recognition engine) and TextToSpeech (a voice synthesis engine).

[2429] Operational Overview

[2430] Profile setup and support

[2431] When a user launches the application for the first time, the server provides a user profile setting screen where the user enters personal information such as name, age, and characteristics, which the device stores locally and then sends to the server.

[2432] The server stores the received information in a database and generates a profile based on the user's characteristics. Based on this profile, the server provides support tailored to the user's characteristics.

[2433] Action plan generation and notification

[2434] When a user inputs their plans for the store (for example, purchasing products or checking in at a service counter) by voice or text, the terminal converts this into text data and sends it to the server.

[2435] The server generates an action plan and related reminders based on the analyzed text data and sends them to the device.

[2436] The device will display a reminder at the specified time, prompting the user to take action.

[2437] Recognizing and responding to emotional states

[2438] The device uses a camera and microphone to collect the user's facial expressions and voice, and inputs this into an emotion recognition engine in real time.

[2439] The server's emotion recognition engine analyzes the user's current emotional state and generates appropriate reminder and support messages based on this. For example, if the user is feeling stressed, it will display a message such as "It's time to take a short break."

[2440] Communication Support

[2441] When a user needs to communicate with others, the user inputs a specific example (for example, the content of a conversation with a store clerk) by voice or text.

[2442] The device analyzes the text data and sends it to the server. The server generates an optimal communication support message based on the analysis results and sends it to the device.

[2443] The terminal displays the generated message to the user, thereby supporting smooth communication.

[2444] Examples and Prompts

[2445] Usage example

[2446] For example, if a user plans to "purchase a product at 1:00 p.m.", the following steps are taken:

[2447] The user speaks, "I'll buy the product at 1pm."

[2448] The terminal converts this into text and sends it to the server.

[2449] The server generates...

Claims

1. A means for accepting input information from a user; means for analyzing the input information and generating an action plan; means for scheduling notifications based on said action plan; means for sending the notification to a terminal; A system including:

2. a means for setting user profile information; means for storing said profile information; A means of providing support tailored to the characteristics of users; The system of claim 1 further comprising:

3. means for converting voice input into text data; means for analyzing the text data and extracting an appropriate action; means for generating a reminder based on the action; means for notifying the reminder; The system of claim 1 further comprising:

4. means for generating a communication support message based on the characteristics of the user; means for providing said message to a user; The system of claim 1 further comprising:

5. means for providing said notification through a smart device; The smart device includes smart glasses, a smartphone, and a wireless earphone; The system of claim 1 further comprising:

Images