System

Abstract

A system is provided.SOLUTION: A system comprising: means for inputting information of a submission material by voice recognition; means for storing the information of the submission material in a database; means for confirming a deadline of the submission material and generating a reminder; means for notifying a user of the reminder; means for inputting a progress status from the user by voice recognition; means for storing the progress status in the database; and means for notifying a sharing target of the progress status.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Many people, especially students and working adults, have difficulty managing submissions and planning their progress. They often forget to submit work, are unable to manage their progress, and suffer from inadequate evaluations and lack of achievement. Current calendar apps and reminder functions require each individual submission to be managed, which is time-consuming and does not provide sufficient progress management or follow-up. Therefore, there is a need for a system that can effectively manage submission deadlines and progress, and provide appropriate reminders and follow-up. [Means for solving the problem]

[0005] The present invention is an AI system that supports the management of submissions, and includes the following means: means for inputting submission information using voice recognition, means for storing the submission information in a database, means for checking submission deadlines and generating reminders, means for notifying users of the reminders, means for users to input progress using voice recognition, means for saving the progress in a database, and means for notifying sharing targets of the progress. This system allows users to efficiently manage submissions, understand progress in real time, and receive appropriate reminders and follow-ups. This reduces the difficulties faced by people who have difficulty managing submissions and provides an environment in which they can be properly evaluated.

[0006] "Submissions" are learning content or work deliverables that users must submit as assignments or projects.

[0007] "Speech recognition" is a technology in which a computer system analyzes what a user says and converts it into text data.

[0008] "Database" means a digital storage system for efficiently managing and storing information about submissions and progress.

[0009] "Reminders" are alert features that notify users of specific deadlines or events.

[0010] "Progress" is the state of how much of an ongoing submission a user has completed.

[0011] "Notifications" are messages that inform users and those they share with important information, such as reminders and progress.

[0012] "Sharing target" refers to the people with whom you share progress and submission information, specifically teachers, friends, colleagues, etc.

[0013] An "AI system" is a system that uses artificial intelligence to automate and efficiently support the management of submissions and progress checks. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention is an AI system that supports submission management and is designed to enable users to effectively manage submissions and understand progress in a timely manner. The following describes an embodiment of the system.

[0036] System Configuration

[0037] This system automates the entire process from users registering submission information by voice input, storing it in a database, generating and notifying reminders at appropriate times, and notifying those sharing the progress.The system is mainly composed of three parties: a server, a terminal, and a user.

[0038] Program processing explanation

[0039] Speak and remember submission information

[0040] When a user inputs information about their submission by voice, the device collects the voice data. The device then uses a voice recognition module to convert this voice data into text data. For example, if a user says, "Register my next math homework," the device responds, "Please tell me the details of the homework," and if the user says, "Complete the problems on pages 10 to 15. The due date is next Monday," the device converts this into text data. This text data is sent from the device to the server.

[0041] Submission information stored in a database

[0042] The server receives the submitted information from the device, takes measures against SQL injection, and stores it in a database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored.

[0043] Check submission deadlines and generate reminders

[0044] The server periodically scans the database to check when assignments are due. When a deadline approaches, it generates a reminder, for example, the day before. The reminder might be something like, "Your math homework is due tomorrow, how's it going?"

[0045] Reminder notifications

[0046] The server generates a reminder and sends it to the device, which receives it and displays it to the user as a push notification, allowing the user to receive timely reminders without forgetting the deadline.

[0047] Voice input and sending of progress

[0048] When a user reports their progress against a reminder by voice, the device collects the voice and converts it into text data using a speech recognition module. For example, if a user says, "I'm only halfway done," the information is sent to the server as text data.

[0049] Save your progress and set next reminders

[0050] The server saves the progress to the database, records it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling it to be generated again on the "morning of the submission date".

[0051] Share your progress

[0052] The server analyzes the progress in real time and notifies the sharing target (teacher or friend). For example, it may notify the progress such as "User A is 50% complete, deadline is tomorrow." The device receives this notification and displays it to the sharing target.

[0053] Specific examples

[0054] Specifically, the user voice-enters information such as "solve the problems on pages 10 to 15" for math homework, and the information is stored in a database. A reminder is then generated the day before the submission deadline and sent to the device. If the user reports their progress as "only half done," the information is recorded in the database again, and the progress is set to 50% complete. The progress is also notified to the teacher, allowing both the user and the teacher to keep track of the progress of the submission.

[0055] In this way, the system combines voice input, database management, reminder functions, progress tracking and sharing functions to streamline the management of submissions.

[0056] The processing flow will be explained below.

[0057] Step 1:

[0058] The user enters the submission information by voice, for example, saying, "AI, register my next math homework."

[0059] Step 2:

[0060] The device collects the voice data and converts it into text data using a voice recognition module. The converted text data becomes "Register my next math homework."

[0061] Step 3:

[0062] The device asks the user to confirm the voice recognition result and ask for additional information. The device asks the user, "Please tell me the details of your homework."

[0063] Step 4:

[0064] The user provides details by voice, for example, "Complete the problems on pages 10 to 15, due next Monday."

[0065] Step 5:

[0066] The device uses a voice recognition module to convert the detailed information into text data, which reads, "Complete the questions on pages 10 to 15. The deadline is next Monday."

[0067] Step 6:

[0068] The terminal parses the converted text data into JSON format or similar and sends it to the server using an HTTP request.

[0069] Step 7:

[0070] The server parses the received data, applies SQL injection protection, and stores it in a database. For example, "Math homework: solve problems on pages 10 to 15, due date: next Monday" is saved.

[0071] Step 8:

[0072] The server periodically scans the database to see when submissions are due, and in this case runs a query to extract submissions that are about to expire.

[0073] Step 9:

[0074] The server generates reminders for upcoming submissions, such as "Your math homework is due tomorrow, how's it going?"

[0075] Step 10:

[0076] The server sends the generated reminder to the device, which then notifies the user of the reminder using the Push notification API.

[0077] Step 11:

[0078] The user receives reminders and reports their progress verbally, for example, "I'm only halfway done."

[0079] Step 12:

[0080] The device collects the voice recording of the progress and converts it into text data using the speech recognition module. The converted text data is "It's only half done."

[0081] Step 13:

[0082] The terminal sends the progress report text data to the server. The data is transferred to the server using an HTTP request.

[0083] Step 14:

[0084] The server stores the received progress report in a database, for example recording the progress as "50% complete."

[0085] Step 15:

[0086] The server will set the next reminder as needed based on the progress, for example, scheduling another reminder to be generated on the morning of the submission date.

[0087] Step 16:

[0088] The server analyzes the progress in real time and notifies the sharing target. For example, it generates information such as "User A is 50% done, deadline is tomorrow."

[0089] Step 17:

[0090] The server sends progress notifications to the recipients (teachers and friends), who receive the notifications via Webhooks or the notification API.

[0091] Step 18:

[0092] The device will receive the notification and display it on the sharing target. For example, the progress will be displayed on the screen of a smartphone or PC.

[0093] In this way, users can effectively manage the progress of their submissions and use the reminder function, and teachers and friends can also grasp the progress of their submissions in real time.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] Conventional submission management systems require users to manually enter submission information, which is cumbersome and time-consuming. Also, deadline reminders and progress management are often done manually, which can lead to forgetting. Furthermore, progress is not shared in real time, which means that stakeholders are unable to grasp the latest information, making it difficult to provide appropriate support and advice.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes means for inputting submission information by voice recognition, means for storing the submission information in a database, means for checking submission deadlines and generating reminders, means for notifying the user of the reminder, means for the user to input progress status by voice recognition, means for saving the progress status in a database, means for notifying sharing targets of the progress status, and means for analyzing the submission information and progress status in real time and setting an appropriate reminder schedule. This allows users to easily manage submissions by voice input, and enables them to remember submission deadlines by receiving reminders and share progress information with relevant parties in real time.

[0099] "Submission information" is detailed data about homework, assignments, documents, etc. that a user must submit.

[0100] "Speech recognition" is a technology that recognizes a user's voice as digital data and converts it into corresponding text data.

[0101] A "database" is a system for efficiently storing, managing, and searching data such as submission information and progress status.

[0102] A "reminder" is a message or alarm that is periodically sent to prompt a user to take a specific action.

[0103] "Progress" is information that indicates the progress and degree of achievement of the submission that the user is working on until completion.

[0104] "Shared with" refers to other users or interested parties (e.g., teachers, friends) who are set up to receive the user's progress information.

[0105] "Real-time analysis" is a technology that analyzes data immediately at the moment it is generated and extracts or processes the necessary information.

[0106] "Setting a schedule" is the act of planning in advance when a specific event or reminder will occur and managing it in the system.

[0107] The present invention is an AI system that supports submission management and is designed to enable users to efficiently manage submissions and keep track of progress in a timely manner. Specific embodiments of this system are described below.

[0108] System Configuration

[0109] This system is primarily composed of three parties: a server, a terminal, and a user, and automates a series of processes including voice input, database management, reminder functions, progress tracking, and sharing functions.

[0110] 1. Voice input of submission information

[0111] The user inputs the information for the assignment by voice. The device (smartphone or PC) collects the voice and converts it into text using a speech recognition module (such as Google Cloud Speech-to-Text API). For example, if the user says, "Register my next math homework," the device responds with, "Please tell me the details of the homework." If the user says, "Complete the problems on pages 10 to 15. The due date is next Monday," the device converts this into text data and sends it to the server.

[0112] 2. Database storage of submission information

[0113] The server receives the submitted information sent from the device, and stores it in a database (e.g., MySQL) after implementing SQL injection protection. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored.

[0114] 3. Check submission deadlines and generate reminders

[0115] The server periodically scans the database for deadlines, and generates reminders as they approach, for example the day before. The reminder might be something like, "Your math homework is due tomorrow, how's it going?"

[0116] 4. Reminder Notifications

[0117] The server generates a reminder and sends it to the device, which receives it and displays it to the user as a push notification, allowing the user to receive timely reminders without forgetting the deadline.

[0118] 5. Voice input and sending of progress

[0119] The user reports their progress against the reminder by voice. For example, they might say, "I'm only halfway done." The device collects the voice and converts it into text using a speech recognition module. The text data is then sent to the server.

[0120] 6. Save your progress and set your next reminder

[0121] The server saves the progress to a database, recording it as "Progress: 50% complete." It then sets the next reminder appropriately, scheduling it to be generated again on the morning of the submission date, for example.

[0122] 7. Share your progress

[0123] The server analyzes the progress in real time and notifies the sharing target (for example, a teacher or friend). For example, information such as "User A is 50% complete, deadline is tomorrow" is notified to the sharing target. The device receives this notification and displays it to the relevant parties.

[0124] Specific examples

[0125] For example, a user might say, "Register my next math homework assignment," and then enter specific details such as, "Complete the problems on pages 10 to 15. Due date: next Monday." This information is stored in the database, and a reminder is generated and sent to the device the day before the deadline. If the user receives the reminder and reports their progress as "only half done," this information is recorded in the database and the progress is marked as 50% complete. This progress status is also notified to the teacher, allowing both the teacher and the user to keep up to date with the latest progress.

[0126] Prompt Sentence Examples

[0127] Here are some example prompts to input to a generative AI model:

[0128] "Sign up my next math homework assignment"

[0129] "Please tell me the details of your homework."

[0130] "Complete the problems on pages 10 to 15. Due next Monday."

[0131] "It's only half done"

[0132] The system automates a series of processes, from voice input to database management, reminder generation, progress tracking and sharing, streamlining the management of submissions.

[0133] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0134] Step 1:

[0135] Dictate submission information

[0136] The user inputs the information for the submission by voice. For example, the user speaks into the microphone of their smartphone or PC, saying, "Register my next math homework." Voice data is acquired as input. The device collects this voice data.

[0137] Specific behavior:

[0138] The user opens the app on their smartphone, presses the "Start Recording" button, and begins speaking. The audio data is input into the device via the microphone.

[0139] Step 2:

[0140] Converting audio data to text

[0141] The device sends the collected voice data to a voice recognition module (for example, Google Cloud Speech-to-Text API) and converts the voice data into text data. Voice data is used as input and text data is generated as output. For example, the voice saying "Please register my next math homework" is converted into text.

[0142] Specific behavior:

[0143] The device sends the voice data to the speech recognition API and displays the returned text. The device prompts again, "Please tell us the details of your homework," and the user responds, "I will complete the problems on pages 10 to 15. The due date is next Monday," and the device converts this into text.

[0144] Step 3:

[0145] Saving submission information

[0146] The terminal sends text data to the server. The server stores the received information in a database (e.g., MySQL) after implementing SQL injection protection measures. Text data is received as input and stored in the database as output.

[0147] Specific behavior:

[0148] The text data "Math homework: solve problems on pages 10 to 15, due date: next Monday" is sent from the terminal to the server, which then stores it in a database.

[0149] Step 4:

[0150] Check the submission deadline

[0151] The server periodically scans the database to check submission deadlines, using the submission information in the database as input and producing a list of submissions that are approaching due dates as output.

[0152] Specific behavior:

[0153] The server checks the database every day at 2:00 PM to find any assignments that are due soon. "I have math homework due next Monday," it says.

[0154] Step 5:

[0155] Reminder generation and notifications

[0156] The server generates reminders for upcoming submissions. For example, the day before a submission is due, it creates a reminder saying, "Tomorrow's math homework is due, how's your progress?" and sends it to the device. The input is a list of upcoming submissions, and the output is a reminder.

[0157] Specific behavior:

[0158] The server generates a reminder "Tomorrow's math homework is due, how's your progress?" and sends it to the device, which displays it to the user as a push notification.

[0159] Step 6:

[0160] Voice input and sending of progress

[0161] The user reports their progress against the reminder by voice, for example, saying "I'm only halfway done." The device collects this voice, converts it into text data using a speech recognition module, and sends it to the server. The voice data of the progress is used as input, and text data is generated as output.

[0162] Specific behavior:

[0163] The user says "I'm only halfway there," and the device converts the speech into text and sends it to the server.

[0164] Step 7:

[0165] Save your progress and set next reminders

[0166] The server saves the progress in a database, for example, recording it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling it to be generated again on the "morning of the submission date". The progress data is received as input, and the next reminder is set as output.

[0167] Specific behavior:

[0168] The server saves "Progress: 50% complete" in the database and sets the next reminder for "the morning of the submission date."

[0169] Step 8:

[0170] Share your progress

[0171] The server analyzes the progress in real time and notifies the sharing target (e.g., teacher or friend). The progress data is used as input and notification data is generated as output.

[0172] Specific behavior:

[0173] The server generates a notification saying "User A is 50% done, deadline is tomorrow" and sends it to the sharing target. The device receives and displays this notification.

[0174] (Application example 1)

[0175] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0176] In factory production line work, workers are required to efficiently manage progress and task status. Conventional methods require a lot of manual input and confirmation work, which not only takes time and effort but also risks progress management errors and delayed reminders. Furthermore, it is difficult to share progress in real time, which can reduce the efficiency of the entire production line. To solve these issues, an efficient and automated progress management system that utilizes voice input is needed.

[0177] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0178] In this invention, the server includes means for inputting information about submissions through voice recognition, means for storing the information about the submissions in a database, means for checking deadlines for submissions and generating reminders, means for notifying users of the reminders, means for users to input progress status through voice recognition, means for saving the progress status in a database, means for notifying sharing targets of the progress status, and means for generating next work instructions based on updates to the progress status. This makes it possible to efficiently manage task progress on production lines in factories and automatically generate and notify reminders. Furthermore, sharing progress status and generating next work instructions in real time can improve the efficiency of the entire production line.

[0179] Below are definitions of important words.

[0180] "Submission information" refers to details of the task or work that a user registers through voice input.

[0181] The "voice recognition means" is a module that has the function of acquiring the words spoken by the user as data and converting the voice data into text data.

[0182] "Means for storing in a database" refers to a device or function that uses a secondary storage device to centrally store information about submissions and progress status.

[0183] The "reminder generator" is a function that automatically creates reminders to notify users at appropriate times based on deadlines for submissions.

[0184] A "means for notifying a reminder" is a device or function that sends the generated reminder to the user via push notification, email, or the like.

[0185] "Progress" refers to information reported by a user about the degree of completion or current progress of a task or submission.

[0186] "Means for inputting progress status by voice recognition" refers to a device or function that allows a user to report progress status by voice and convert it into text data.

[0187] The "means for generating the next work instruction" is a function that automatically determines the next task and work content to be done based on the current progress status and generates instructions.

[0188] The "means for notifying the sharing target of the progress status" is a device or function for reporting the progress status to the sharing target (such as a superior or colleague) in real time.

[0189] This invention is a system for efficiently managing task progress on a production line in a factory. Specifically, it improves production efficiency by registering task details and deadlines through voice input, saving them in a database, and managing the progress in real time.

[0190] System Configuration

[0191] This system mainly consists of a user, a device (such as a smartphone), and a server. The user inputs task details and deadlines by voice via their smartphone, and this data is converted into text data through a voice recognition module. The voice recognition module can be, for example, the SpeechRecognition library.

[0192] Speak and remember submission information

[0193] When a user uses their smartphone to say, "Please add a new task," the device collects voice data through the microphone. At this time, the device uses a voice recognition module to convert the collected voice data into text data. For example, if a user says, "In the next process, assemble 10 parts. The deadline is next Monday," the information is converted into text and stored in a database.

[0194] Submission information stored in a database

[0195] The server receives task information sent from the terminal and stores it in a database after implementing security measures. This system uses SQLite to store information. For example, information such as "Part assembly: Assemble 10 parts, Deadline: Next Monday" is stored.

[0196] Check submission deadlines and generate reminders

[0197] The server periodically scans the database to check deadlines for submissions. When a deadline approaches, it generates a reminder, for example, the day before the deadline. The reminder might say something like, "Tomorrow's task 'Assemble 10 parts' is due, please check your progress." This reminder is sent to the user as a push notification.

[0198] Voice input and sending of progress

[0199] After receiving the reminder, the user can report their progress by voice, for example, by saying "I'm only halfway done," and the device will convert that voice into text data and send it to the server.

[0200] Save your progress and set next reminders

[0201] The server records the progress in a database, saving it as "Progress: 50% complete", and sets the next reminder if necessary, for example, scheduling another reminder to be generated "on the morning of the submission date".

[0202] Share your progress

[0203] The server analyzes the progress in real time and notifies the sharing target (such as a superior or colleague). For example, progress can be shared in the form of "Progress: User A is 50% complete, deadline is tomorrow."

[0204] Specific examples

[0205] For example, worker A might say to his smartphone, "Please add the task of assembling parts," and then voice-input, "Assemble 10 parts. Deadline is next Monday." This information is converted into text data and stored in a database. The day before the deadline, a push notification arrives saying, "Task deadline is approaching," and if worker A reports, "It's only half done," the progress is recorded as 50%. The supervisor can then keep track of the progress in real time.

[0206] Prompt Sentence Examples

[0207] "Please add a task for the next production line: Please tell us the details of the task."

[0208] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0209] Step 1: User dictates task information

[0210] The user uses the smartphone's microphone to input task details and deadlines by voice. For example, when the user says, "Add a new task," the device collects the voice data, and the user utters, "In the next process, assemble 10 parts. The deadline is next Monday." The input data is voice data, which is converted into text data in the next step.

[0211] Step 2: The device converts the audio data into text data.

[0212] A speech recognition module (e.g., SpeechRecognition library) installed on the device acquires voice data and converts it into text data. The input is the user's voice data, and the output is text data. Specifically, the text data obtained is "In the next process, assemble 10 parts. The deadline is next Monday."

[0213] Step 3: The server saves the text data to the database

[0214] The server receives text data sent from the terminal and stores it in a database. The input is text data, and the output is a record stored in the database. The server stores information such as "Part assembly: Assemble 10 parts, Deadline: Next Monday" while implementing security measures.

[0215] Step 4: The server checks the submission deadline and generates a reminder

[0216] The server periodically scans the database to check deadlines for submissions. The input is the deadline information in the database, and the output is a reminder. When the deadline approaches (e.g., the day before the deadline), the server generates a reminder saying, "Tomorrow's task 'Assemble 10 parts' is due, please check your progress."

[0217] Step 5: The device notifies the user of the reminder

[0218] The server sends the generated reminder to the device, and the device notifies the user via a push notification. The input is the generated reminder, and the output is a notification displayed on the user's smartphone. Specifically, a notification appears on the user's screen saying, "Tomorrow's task 'Assemble 10 parts' is due. Please check your progress."

[0219] Step 6: User dictates progress

[0220] After receiving the reminder, the user reports their progress by voice, for example, saying, "I'm only halfway done." The input is the user's voice data, which is converted to text data in the next step.

[0221] Step 7: The device converts the progress voice data into text data.

[0222] The speech recognition module converts the user's voice data into text data. The input is the voice data of the progress, and the output is text data. Specifically, the text data obtained is "It's only half done."

[0223] Step 8: The server saves the progress to a database

[0224] The server receives the text data of the progress status sent from the device and stores it in the database. The input is the text data of the progress status, and the output is a record stored in the database. Specifically, it is stored as "Progress: 50% complete."

[0225] Step 9: The server notifies the sharer of its progress

[0226] The server analyzes the progress in real time and notifies the sharing target (superior or colleague). The input is the progress data, and the output is a notification message. Specifically, it is shared as "Progress content: User A is 50% progress, deadline is tomorrow."

[0227] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0228] The present invention combines an AI system that supports submission management with an emotion engine, and is a system that recognizes the user's emotions and can more effectively manage submissions and notify progress based on those emotions. The following describes specific embodiments of the system.

[0229] System Configuration

[0230] This system allows users to input information about submissions by voice, stores it in a database, generates and notifies reminders at appropriate times, and notifies sharing targets of progress.In addition, it recognizes the user's emotions and adjusts the content of the dialogue and reminders.The system is mainly composed of four parties: the server, the terminal, the user, and the emotion engine.

[0231] Program processing explanation

[0232] Speak and remember submission information

[0233] When a user enters information about their assignment by voice, the device collects the voice data. The device then converts the voice data into text data using a voice recognition module. For example, if the user says, "Register my next math homework," the device responds with, "Please tell me the details of the homework," and if the user says, "Please solve the problems on pages 10 to 15. The due date is next Monday," the device converts the data into text data. This text data is then sent from the device to the server.

[0234] Submission information stored in a database

[0235] The server receives the submitted information from the device, takes measures against SQL injection, and stores it in a database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored.

[0236] Check submission deadlines and generate reminders

[0237] The server periodically scans the database to check when assignments are due. When a deadline approaches, it generates a reminder, for example, the day before. The reminder might be something like, "Your math homework is due tomorrow, how's it going?"

[0238] Reminder notifications

[0239] The server generates a reminder and sends it to the device. The device receives the reminder and displays it to the user as a push notification. This allows the user to receive reminders at the appropriate time without forgetting the submission deadline.

[0240] Voice input and sending of progress

[0241] The user reports their progress against the reminder by voice. For example, if they say, "I'm only halfway done," the device collects the voice and converts it into text data using a speech recognition module. The converted text data becomes, "I'm only halfway done."

[0242] Save your progress and set next reminders

[0243] The server saves the progress to the database, recording it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling another reminder to be generated "on the morning of the submission date".

[0244] Share your progress

[0245] The server analyzes the progress in real time and notifies the sharing target (teacher or friend). For example, it may notify the progress such as "User A is 50% complete, deadline is tomorrow." The device receives this notification and displays it to the sharing target.

[0246] Implementing the Emotion Engine

[0247] The emotion engine is a module that analyzes the user's voice data and determines their emotional state. When a user voice-enters information about a submission or reports progress, the engine analyzes the voice data to recognize the user's emotional state.

[0248] Emotion-aware reminder adjustment

[0249] The server adjusts the content and timing of reminders based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, the server may adjust the content and timing of reminders to be softer or to send them less frequently.

[0250] Customize notifications with emotion recognition

[0251] The server customizes the progress notification content based on the user's emotions recognized by the emotion engine. For example, if the user is feeling impatient, the notification content can be adjusted by adding an encouraging comment.

[0252] Specific examples

[0253] Specifically, a user can voice-input information such as "solve the problems on pages 10 to 15" for math homework, and that information is stored in a database. A reminder is then generated the day before the deadline and sent to the device. If the emotion engine recognizes impatience or anxiety when the user reports "only halfway done," the server adds a comforting message to the next reminder: "Keep up the good work, you're almost there!" The progress is also recognized as 50% complete, and the sharing target is notified of this information.

[0254] In this way, this system combines voice input, database management, reminder functions, progress tracking and sharing functions, as well as an emotion engine to streamline the management of submissions and provide support that is sensitive to the user's emotions.

[0255] The processing flow will be explained below.

[0256] Step 1:

[0257] The user enters the submission information by voice, for example, saying, "AI, register my next math homework."

[0258] Step 2:

[0259] The device collects the voice data and converts it into text data using a voice recognition module. The converted text data becomes "Register my next math homework."

[0260] Step 3:

[0261] The device asks the user to confirm the voice recognition result and ask for additional information. The device asks the user, "Please tell me the details of your homework."

[0262] Step 4:

[0263] The user provides details by voice, for example, "Complete the problems on pages 10 to 15, due next Monday."

[0264] Step 5:

[0265] The device uses a voice recognition module to convert the detailed information into text data, which reads, "Complete the questions on pages 10 to 15. The deadline is next Monday."

[0266] Step 6:

[0267] The terminal parses the converted text data into JSON format or similar and sends it to the server using an HTTP request.

[0268] Step 7:

[0269] The server parses the received data, applies SQL injection protection, and stores it in a database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is saved.

[0270] Step 8:

[0271] The server periodically scans the database to see when submissions are due, and in this case runs a query to extract submissions that are about to expire.

[0272] Step 9:

[0273] The server generates reminders for upcoming submissions, such as "Your math homework is due tomorrow, how's it going?"

[0274] Step 10:

[0275] The server sends the generated reminder to the device, which then notifies the user of the reminder using the Push notification API.

[0276] Step 11:

[0277] The user receives reminders and reports their progress verbally, for example, "I'm only halfway done."

[0278] Step 12:

[0279] The device collects the voice recording of the progress and converts it into text data using the speech recognition module. The converted text data is "It's only half done."

[0280] Step 13:

[0281] The terminal sends the converted text data to the server, and transfers the data to the server using an HTTP request.

[0282] Step 14:

[0283] The server stores the received progress report in a database, for example recording the progress as "50% complete."

[0284] Step 15:

[0285] The server will set the next reminder as needed based on the progress, for example, scheduling another reminder to be generated on the morning of the submission date.

[0286] Step 16:

[0287] The server analyzes the progress in real time and notifies the sharing target. For example, it generates information such as "User A is 50% done, deadline is tomorrow."

[0288] Step 17:

[0289] The server sends progress notifications to the recipients (teachers and friends), who receive the notifications via Webhooks or the notification API.

[0290] Step 18:

[0291] The device will receive the notification and display it on the sharing target. For example, the progress will be displayed on the screen of a smartphone or PC.

[0292] Step 19:

[0293] When a user expresses emotion during a reminder or report, the device sends the voice data to the emotion engine.

[0294] Step 20:

[0295] The emotion engine analyzes the voice data and recognizes the user's emotional state, for example, recognizing that the user is feeling impatient.

[0296] Step 21:

[0297] The server adjusts the content and timing of reminders based on the analysis results of the emotion engine, for example, by making the reminder content softer or reducing the frequency of reminders.

[0298] Step 22:

[0299] The server customizes the progress notification content based on the analysis results of the emotion engine, for example by adding encouraging comments.

[0300] In this way, a system that combines an emotion engine makes it possible to manage submissions in accordance with the user's emotions, thereby providing more personalized support.

[0301] Example 2

[0302] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0303] Conventional submission management systems lack support that takes into account the user's emotional state, which often leads to situations where users feel stressed or anxious. Furthermore, users' progress is rarely shared or notified in real time, making it difficult to respond in a timely manner. This results in inefficient submission management and increases the burden on users.

[0304] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for inputting information about a submission by voice recognition, means for storing information about the submission in a database, means for checking the deadline for the submission and generating a reminder, means for notifying the user of the reminder, means for the user to input progress by voice recognition, means for saving the progress in a database, means for notifying sharing targets of the progress, means for analyzing the user's emotions and adjusting the reminder, and means for analyzing the user's emotions and customizing the notification content. This makes it possible to efficiently manage submissions and provide support that is sensitive to the user's emotions.

[0305] "Submission Information" means detailed information about the work or task that a User is required to submit, including the assignment content, due date, and related materials.

[0306] "Speech recognition" refers to the technology that analyzes a user's voice and converts it into text data, making it possible to use voice as an input method.

[0307] "Database" refers to a collection of electronic records that systematically organizes and manages information for rapid retrieval and access. In this context, it is used to store submission information, progress status, etc.

[0308] A "reminder" is a notification or warning message that prompts a user to take a specific action, helping users to remember deadlines for submissions and other matters.

[0309] "Notification" refers to the process by which a system communicates important information to users based on pre-set timing and conditions. Notifications are given via methods such as push notifications and emails.

[0310] "Progress" refers to the work or task execution status of a user. In this case, it shows how much of a particular task has been completed.

[0311] "SQL injection countermeasures" refers to security measures to prevent unauthorized access to a database, thereby ensuring the safety of the database.

[0312] "Emotion engine" refers to technology that analyzes the user's voice data and recognizes the emotions contained within it, enabling customization based on the user's psychological state.

[0313] "Speech recognition module" refers to a combination of hardware and software for analyzing voice data and converting it into text data. Specifically, this includes the use of APIs.

[0314] A "cron job" is a scheduling technique used to automate periodic tasks within a system, such as scanning a database at regular intervals or generating reminders.

[0315] "Firebase Cloud Messaging" refers to a cloud service for sending notifications to mobile and web applications, enabling real-time push notifications.

[0316] "Microsoft Azure Cognitive Services" refers to a collection of cloud-based artificial intelligence services that provide capabilities such as natural language processing, image recognition, and speech recognition, which are used in this case for sentiment analysis.

[0317] "Emotion API" refers to an interface for detecting and analyzing emotions from a user's voice and images, making it possible to recognize the user's emotional state.

[0318] The present invention combines an AI system that supports submission management with an emotion engine, and is a system that recognizes the user's emotions and can more effectively manage submissions and notify progress based on those emotions. The following describes specific embodiments of the system.

[0319] System Configuration

[0320] This system is mainly composed of four elements: a server, a terminal, a user, and an emotion engine. The system allows users to input information about their submissions by voice, stores it in a database, generates and notifies reminders at appropriate times, and notifies the sharing target of progress. In addition, the system recognizes the user's emotions and adjusts the content of the dialogue and reminders.

[0321] Speak and remember submission information

[0322] When a user voice-inputs information about their assignment, the device collects the voice data. The device uses the Google Cloud Speech-to-Text API as a voice recognition module to convert the voice data into text data. For example, if a user says, "Register my next math homework," the device responds with, "Please tell me the details of the homework," and if the user verbally replies, "I will solve the problems on pages 10 to 15. The due date is next Monday," the content is converted into text data. This text data is sent from the device to the server.

[0323] Submission information stored in a database

[0324] The server receives the submitted information from the device, takes measures against SQL injection, and stores it in a MySQL database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored in the database.

[0325] Check submission deadlines and generate reminders

[0326] The server periodically uses a cron job to scan the database and check deadlines for submissions. When a deadline approaches, it generates a reminder, for example the day before. The reminder might be something like "Your math homework is due tomorrow, how's it going?"

[0327] Reminder notifications

[0328] The server generates a reminder and sends it to the device. The device receives the reminder using Firebase Cloud Messaging and displays it to the user as a push notification. This allows users to receive reminders at the appropriate time without forgetting the submission deadline.

[0329] Speech to text progress

[0330] The user reports their progress against the reminder by voice. For example, if they say, "I'm only halfway done," the device collects the voice and converts it into text data using a speech recognition module. The converted text data is "I'm only halfway done." This text data is then sent back to the server.

[0331] Save your progress and set next reminders

[0332] The server saves the progress to the database, recording it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling another reminder to be generated "on the morning of the submission date".

[0333] Share your progress

[0334] The server analyzes the progress in real time and notifies the sharing target (teacher or friend). For example, it may notify the progress such as "User A is 50% done, deadline is tomorrow." The notification is sent via email or a dedicated application and displayed to the sharing target.

[0335] Implementing the Emotion Engine

[0336] The emotion engine uses the Emotion API of Microsoft Azure Cognitive Services to analyze the user's voice data and determine their emotional state. When a user dictates information for submissions or reports progress, the engine recognizes the user's emotional state by analyzing the voice data.

[0337] Emotion-aware reminder adjustment

[0338] The server adjusts the content and timing of reminders based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, the server may adjust the content and timing of reminders to be softer or to send them less frequently.

[0339] Customize notifications with emotion recognition

[0340] The server customizes the progress notification content based on the user's emotions recognized by the emotion engine. For example, if the user is feeling impatient, the server can adjust the notification content by adding an encouraging comment.

[0341] Specific examples

[0342] Specifically, a user can voice-input information such as "solve the problems on pages 10 to 15" for math homework, and that information is stored in a database. A reminder is then generated the day before the deadline and sent to the device. If the emotion engine recognizes impatience or anxiety when the user reports "only halfway done," the server adds a comforting message to the next reminder: "Keep up the good work, you're almost there!" The progress is also recognized as 50% complete, and the sharing target is notified of this information.

[0343] Examples of prompt statements

[0344] An example of a prompt that can be input to a generative AI model is a user's voice input such as "Please register my next math homework." Another example of a prompt is a user's voice input of their progress, such as "I'm only halfway done."

[0345] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0346] Step 1:

[0347] The user inputs the information about the assignment by voice. For example, when the user says, "Register my next math homework," the voice data is input into the terminal. The terminal collects this voice data.

[0348] Step 2:

[0349] The device converts the collected voice data into text using the Google Cloud Speech-to-Text API, which then provides instructions such as "Register your next math homework assignment."

[0350] Step 3:

[0351] The device sends the text data to the server, which then generates a response message including a follow-up question, such as "Please tell me the details of the next math homework assignment," and sends it back to the device.

[0352] Step 4:

[0353] The device receives the response message from the server and prompts the user by voice, "Please tell me the details of your homework." The user then enters the details by voice, saying, "Please solve the problems on pages 10 to 15. The deadline is next Monday," and the device collects the voice data again.

[0354] Step 5:

[0355] The device converts the collected detailed voice data into text using the Google Cloud Speech-to-Text API, resulting in text such as "Complete the questions on pages 10 to 15. The deadline is next Monday."

[0356] Step 6:

[0357] The device then sends the text data back to the server. The server receives this information, applies SQL injection protection, and stores it in a MySQL database. The stored content is "Math homework: solve problems on pages 10 to 15, due date: next Monday."

[0358] Step 7:

[0359] The server periodically scans the database using a cron job to check deadlines for work, for example, if "Math Homework" is due soon, and generates a reminder if necessary: ​​"Your math homework is due tomorrow, how's it going?"

[0360] Step 8:

[0361] The server generates a reminder and sends it to the device. The device receives the reminder using Firebase Cloud Messaging and sends a push notification to the user saying, "Tomorrow's math homework is due. How's the progress?"

[0362] Step 9:

[0363] The user reports their progress by voice. For example, if they report "I'm only halfway done," the device collects that voice and converts it into text data, "I'm only halfway done," using the Google Cloud Speech-to-Text API.

[0364] Step 10:

[0365] The device sends the text data to the server, which records the progress status in the database as "Progress: 50% complete" and sets the next reminder based on the progress data. For example, it schedules another reminder to be generated on the "morning of the submission date."

[0366] Step 11:

[0367] The server analyzes the progress and notifies the sharing target, such as teachers and friends, of the progress information. For example, it may notify users via email or a dedicated application that "User A is 50% complete, deadline is tomorrow."

[0368] Step 12:

[0369] To recognize the user's emotional state as input, the device uses an emotion engine (Microsoft Azure Cognitive Services' Emotion API) to analyze the user's voice data. Through the analysis, it can determine, for example, whether the user is feeling stressed.

[0370] Step 13:

[0371] The server adjusts the content and timing of reminders based on the emotion recognition results. If the user is feeling stressed, the content of the reminders will be softened or the frequency will be reduced.

[0372] Step 14:

[0373] The server customizes the progress notification content based on the emotion recognition results. For example, if the user is feeling impatient, it adds an encouraging comment such as "Keep up the good work, you're almost there!"

[0374] Examples:

[0375] When a user says, "Sign up my next math homework," the device collects the speech, converts it into text data using a speech recognition module, and sends it to the server. The server responds, and after the user enters details, the server uses an emotion engine to recognize the user's stress and adjusts the content of reminders and notifications.

[0376] (Application example 2)

[0377] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0378] Conventional submission management systems provide reminders and notifications without considering the user's emotional state, resulting in a lack of support appropriate to the user's situation and feelings. Furthermore, they lacked an emotion-based product recommendation function, which meant that the user's shopping experience could not be optimized.

[0379] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting information about a submission using voice recognition, means for storing information about the submission in a database, means for checking the deadline for the submission and generating a reminder, means for notifying the user of the reminder, means for the user to input progress status using voice recognition, means for saving the progress status in a database, means for notifying sharing targets of the progress status, means for recognizing the user's emotions from voice and facial expressions, means for adjusting the content and timing of the reminder based on the emotion, and means for recommending products based on the emotion. This enables submission management that is sensitive to the user's emotions and product recommendations based on emotions.

[0380] A "submittable" is a completed product or something to be prepared related to some task or activity.

[0381] "Speech recognition" is a technology that analyzes speech and converts it into text or commands.

[0382] A "database" is a collection of information that is structured so that the data can be efficiently managed and searched.

[0383] A "reminder" is a message sent to a user to notify them of a specific date, time, or event.

[0384] "Progress" is information that indicates the degree of completion or current status of a specific task or work.

[0385] A "shared party" is a person or organization with whom specific data or information is shared.

[0386] "Emotion recognition" is a technology that identifies a user's emotional state by analyzing their voice and facial expressions.

[0387] "Product recommendation" refers to the selection of products or services suggested for purchase based on the user's preferences and status.

[0388] The present invention combines a system for supporting submission management with an emotion engine, and can provide reminders and product recommendations according to the user's emotional state. An embodiment of the present invention will be described in detail below.

[0389] System Configuration

[0390] This system mainly consists of four components: a server, a terminal, a user, and an emotion engine.

[0391] The server manages submission information, checks deadlines, generates and notifies reminders, manages progress, recognizes emotions, and recommends products.

[0392] The device, typically a smartphone or tablet, accepts voice input, converts the voice data into text, and displays reminders and notifications to the user.

[0393] Users provide submission information and progress to the system through voice input and receive reminders and notifications.

[0394] The emotion engine analyzes voice and facial expression data to recognize the user's emotional state, and customizes reminder content and product recommendations based on this emotion recognition.

[0395] Hardware and Software

[0396] The main hardware and software used in this system are as follows:

[0397] Hardware: Smartphone (microphone, camera), server

[0398] Software: speech recognition modules (e.g., Google Speech Recognition API), emotion recognition systems (e.g., Emotion Recognizer), database management systems (e.g., SQLite), notification systems

[0399] Processing flow

[0400] 1. Voice to text conversion:

[0401] Users can provide information about their submissions and progress by speaking into the device. This voice data is collected through the smartphone's microphone and converted into text data using a voice recognition module.

[0402] 2. Database storage:

[0403] Submission information and progress status converted into text data are sent from the device to a server and securely stored in a database management system (e.g., SQLite).

[0404] 3. Emotion recognition:

[0405] The user's voice and facial expression data are analyzed through an emotion engine to recognize their emotional state, and the emotion recognition results are sent to the server in real time.

[0406] 4. Reminder generation and notifications:

[0407] The server checks the deadline for submissions and generates reminders. The content and timing of the reminders are adjusted based on the emotion recognition results. For example, if the user is feeling stressed, the reminder content will be softened. The reminder is then sent to the user's device via a notification system.

[0408] 5. Progress Management and Sharing:

[0409] The user reports their progress by voice, which is stored in a database. The server analyzes the progress in real time and notifies co-users (e.g., teachers, friends).

[0410] 6. Product recommendation:

[0411] Based on the emotion recognition results, the server recommends appropriate products to the user. For example, if the user is feeling stressed, it recommends relaxation products.

[0412] Specific use cases

[0413] For example, a user can say "I want to buy milk" into their smartphone, and the app will add it to the list. It will also send a reminder one week later. Because the user is feeling stressed, the app will recommend products that will help them relax and display a reminder with an encouraging message.

[0414] Prompt Sentence Examples

[0415] "Create a list of products input by the user's voice and recommend products based on their emotions. Build a system that adjusts the content of reminders and notifies them at the appropriate time based on the emotion recognition results."

[0416] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0417] Step 1:

[0418] The user speaks into the device to input information about their submission and progress. The device collects this voice data and converts it into text data using a voice recognition module (e.g., Google Speech Recognition API). For example, if the user speaks, "I want to buy milk," the voice recognition module generates the text data, "I want to buy milk." Input: Voice data, Output: Text data.

[0419] Step 2:

[0420] The terminal sends the generated text data to the server, which stores the submitted work and progress information in a database management system (e.g., SQLite). Input: Text data, Output: Database storage.

[0421] Step 3:

[0422] The user expresses their emotions for the day through their voice and facial expression on the device. The device collects the voice and facial expression data and analyzes their emotional state using an emotion recognition system (e.g., Emotion Recognizer). For example, if the user's voice sounds tense, the emotion recognition system will recognize it as "stress." Input: Voice and facial expression data, Output: Emotional state data.

[0423] Step 4:

[0424] The server periodically scans the submission deadlines stored in the database. When a deadline approaches, it generates a reminder. It adjusts the content and timing of the reminder based on emotion recognition data. For example, if the user is feeling stressed, it generates a gentle reminder such as "The deadline to buy milk is tomorrow. Please stay calm and proceed." Input: submission data, emotional state data, output: reminder data.

[0425] Step 5:

[0426] The server sends the generated reminder data to the device. The device displays the reminder to the user through the notification system, allowing the user to be notified of deadlines and shopping list items at the appropriate time. Input: Reminder data, Output: Reminder notification.

[0427] Step 6:

[0428] The user reports their progress by voice input. The device collects the voice data and converts it into text data using a voice recognition module, just as in step 1. For example, if the user says, "It's only half done," the text data "It's only half done" is generated. Input: Voice data, Output: Text data.

[0429] Step 7:

[0430] The terminal sends the generated text data of the progress to the server. The server saves the progress in a database and analyzes the progress in real time. Input: Text data of the progress, Output: Database storage and analysis results.

[0431] Step 8:

[0432] The server notifies the sharing target (e.g., teacher, friend) of the results of the progress analysis. For example, if a user's progress is 50% complete, a notification stating "User A is 50% complete, deadline is tomorrow" is generated and sent to the sharing target. Input: Progress analysis data, Output: Sharing notification data.

[0433] Step 9:

[0434] The server recommends products to the user based on the emotion recognition results. For example, if the user is feeling stressed, it will recommend relaxation items. Based on the emotion data, the product recommendation algorithm selects the most suitable product and notifies the user. Input: Emotional state data, product database, Output: Product recommendation data.

[0435] Step 10:

[0436] The server sends the recommended product information to the terminal, which then notifies and displays the product information to the user. This makes it easier for the user to purchase products that match their emotional state. Input: product recommendation data, Output: product recommendation notification.

[0437] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0438] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0439] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0440] [Second embodiment]

[0441] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0442] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0443] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0444] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0445] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0446] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0447] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0448] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0449] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0450] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0451] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0452] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0453] The present invention is an AI system that supports submission management and is designed to enable users to effectively manage submissions and understand progress in a timely manner. The following describes an embodiment of the system.

[0454] System Configuration

[0455] This system automates the entire process from users registering submission information by voice input, storing it in a database, generating and notifying reminders at appropriate times, and notifying those sharing the progress.The system is mainly composed of three parties: a server, a terminal, and a user.

[0456] Program processing explanation

[0457] Speak and remember submission information

[0458] When a user inputs information about their submission by voice, the device collects the voice data. The device then uses a voice recognition module to convert this voice data into text data. For example, if a user says, "Register my next math homework," the device responds, "Please tell me the details of the homework," and if the user says, "Complete the problems on pages 10 to 15. The due date is next Monday," the device converts this into text data. This text data is sent from the device to the server.

[0459] Submission information stored in a database

[0460] The server receives the submitted information from the device, takes measures against SQL injection, and stores it in a database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored.

[0461] Check submission deadlines and generate reminders

[0462] The server periodically scans the database to check when assignments are due. When a deadline approaches, it generates a reminder, for example, the day before. The reminder might be something like, "Your math homework is due tomorrow, how's it going?"

[0463] Reminder notifications

[0464] The server generates a reminder and sends it to the device, which receives it and displays it to the user as a push notification, allowing the user to receive timely reminders without forgetting the deadline.

[0465] Voice input and sending of progress

[0466] When a user reports their progress against a reminder by voice, the device collects the voice and converts it into text data using a speech recognition module. For example, if a user says, "I'm only halfway done," the information is sent to the server as text data.

[0467] Save your progress and set next reminders

[0468] The server saves the progress to the database, records it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling it to be generated again on the "morning of the submission date".

[0469] Share your progress

[0470] The server analyzes the progress in real time and notifies the sharing target (teacher or friend). For example, it may notify the progress such as "User A is 50% complete, deadline is tomorrow." The device receives this notification and displays it to the sharing target.

[0471] Specific examples

[0472] Specifically, the user voice-enters information such as "solve the problems on pages 10 to 15" for math homework, and the information is stored in a database. A reminder is then generated the day before the submission deadline and sent to the device. If the user reports their progress as "only half done," the information is recorded in the database again, and the progress is set to 50% complete. The progress is also notified to the teacher, allowing both the user and the teacher to keep track of the progress of the submission.

[0473] In this way, the system combines voice input, database management, reminder functions, progress tracking and sharing functions to streamline the management of submissions.

[0474] The processing flow will be explained below.

[0475] Step 1:

[0476] The user enters the submission information by voice, for example, saying, "AI, register my next math homework."

[0477] Step 2:

[0478] The device collects the voice data and converts it into text data using a voice recognition module. The converted text data becomes "Register my next math homework."

[0479] Step 3:

[0480] The device asks the user to confirm the voice recognition result and ask for additional information. The device asks the user, "Please tell me the details of your homework."

[0481] Step 4:

[0482] The user provides details by voice, for example, "Complete the problems on pages 10 to 15, due next Monday."

[0483] Step 5:

[0484] The device uses a voice recognition module to convert the detailed information into text data, which reads, "Complete the questions on pages 10 to 15. The deadline is next Monday."

[0485] Step 6:

[0486] The terminal parses the converted text data into JSON format or similar and sends it to the server using an HTTP request.

[0487] Step 7:

[0488] The server parses the received data, applies SQL injection protection, and stores it in a database. For example, "Math homework: solve problems on pages 10 to 15, due date: next Monday" is saved.

[0489] Step 8:

[0490] The server periodically scans the database to see when submissions are due, and in this case runs a query to extract submissions that are about to expire.

[0491] Step 9:

[0492] The server generates reminders for upcoming submissions, such as "Your math homework is due tomorrow, how's it going?"

[0493] Step 10:

[0494] The server sends the generated reminder to the device, which then notifies the user of the reminder using the Push notification API.

[0495] Step 11:

[0496] The user receives reminders and reports their progress verbally, for example, "I'm only halfway done."

[0497] Step 12:

[0498] The device collects the voice recording of the progress and converts it into text data using the speech recognition module. The converted text data is "It's only half done."

[0499] Step 13:

[0500] The terminal sends the progress report text data to the server. The data is transferred to the server using an HTTP request.

[0501] Step 14:

[0502] The server stores the received progress report in a database, for example recording the progress as "50% complete."

[0503] Step 15:

[0504] The server will set the next reminder as needed based on the progress, for example, scheduling another reminder to be generated on the morning of the submission date.

[0505] Step 16:

[0506] The server analyzes the progress in real time and notifies the sharing target. For example, it generates information such as "User A is 50% done, deadline is tomorrow."

[0507] Step 17:

[0508] The server sends progress notifications to the recipients (teachers and friends), who receive the notifications via Webhooks or the notification API.

[0509] Step 18:

[0510] The device will receive the notification and display it on the sharing target. For example, the progress will be displayed on the screen of a smartphone or PC.

[0511] In this way, users can effectively manage the progress of their submissions and use the reminder function, and teachers and friends can also grasp the progress of their submissions in real time.

[0512] Example 1

[0513] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0514] Conventional submission management systems require users to manually enter submission information, which is cumbersome and time-consuming. Also, deadline reminders and progress management are often done manually, which can lead to forgetting. Furthermore, progress is not shared in real time, which means that stakeholders are unable to grasp the latest information, making it difficult to provide appropriate support and advice.

[0515] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0516] In this invention, the server includes means for inputting submission information by voice recognition, means for storing the submission information in a database, means for checking submission deadlines and generating reminders, means for notifying the user of the reminder, means for the user to input progress status by voice recognition, means for saving the progress status in a database, means for notifying sharing targets of the progress status, and means for analyzing the submission information and progress status in real time and setting an appropriate reminder schedule. This allows users to easily manage submissions by voice input, and enables them to remember submission deadlines by receiving reminders and share progress information with relevant parties in real time.

[0517] "Submission information" is detailed data about homework, assignments, documents, etc. that a user must submit.

[0518] "Speech recognition" is a technology that recognizes a user's voice as digital data and converts it into corresponding text data.

[0519] A "database" is a system for efficiently storing, managing, and searching data such as submission information and progress status.

[0520] A "reminder" is a message or alarm that is periodically sent to prompt a user to take a specific action.

[0521] "Progress" is information that indicates the progress and degree of achievement of the submission that the user is working on until completion.

[0522] "Shared with" refers to other users or interested parties (e.g., teachers, friends) who are set up to receive the user's progress information.

[0523] "Real-time analysis" is a technology that analyzes data immediately at the moment it is generated and extracts or processes the necessary information.

[0524] "Setting a schedule" is the act of planning in advance when a specific event or reminder will occur and managing it in the system.

[0525] The present invention is an AI system that supports submission management and is designed to enable users to efficiently manage submissions and keep track of progress in a timely manner. Specific embodiments of this system are described below.

[0526] System Configuration

[0527] This system is primarily composed of three parties: a server, a terminal, and a user, and automates a series of processes including voice input, database management, reminder functions, progress tracking, and sharing functions.

[0528] 1. Voice input of submission information

[0529] The user inputs the information for the assignment by voice. The device (smartphone or PC) collects the voice and converts it into text using a speech recognition module (such as Google Cloud Speech-to-Text API). For example, if the user says, "Register my next math homework," the device responds with, "Please tell me the details of the homework." If the user says, "Complete the problems on pages 10 to 15. The due date is next Monday," the device converts this into text data and sends it to the server.

[0530] 2. Database storage of submission information

[0531] The server receives the submitted information sent from the device, and stores it in a database (e.g., MySQL) after implementing SQL injection protection. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored.

[0532] 3. Check submission deadlines and generate reminders

[0533] The server periodically scans the database for deadlines, and generates reminders as they approach, for example the day before. The reminder might be something like, "Your math homework is due tomorrow, how's it going?"

[0534] 4. Reminder Notifications

[0535] The server generates a reminder and sends it to the device, which receives it and displays it to the user as a push notification, allowing the user to receive timely reminders without forgetting the deadline.

[0536] 5. Voice input and sending of progress

[0537] The user reports their progress against the reminder by voice. For example, they might say, "I'm only halfway done." The device collects the voice and converts it into text using a speech recognition module. The text data is then sent to the server.

[0538] 6. Save your progress and set your next reminder

[0539] The server saves the progress to a database, recording it as "Progress: 50% complete." It then sets the next reminder appropriately, scheduling it to be generated again on the morning of the submission date, for example.

[0540] 7. Share your progress

[0541] The server analyzes the progress in real time and notifies the sharing target (for example, a teacher or friend). For example, information such as "User A is 50% complete, deadline is tomorrow" is notified to the sharing target. The device receives this notification and displays it to the relevant parties.

[0542] Specific examples

[0543] For example, a user might say, "Register my next math homework assignment," and then enter specific details such as, "Complete the problems on pages 10 to 15. Due date: next Monday." This information is stored in the database, and a reminder is generated and sent to the device the day before the deadline. If the user receives the reminder and reports their progress as "only half done," this information is recorded in the database and the progress is marked as 50% complete. This progress status is also notified to the teacher, allowing both the teacher and the user to keep up to date with the latest progress.

[0544] Prompt Sentence Examples

[0545] Here are some example prompts to input to a generative AI model:

[0546] "Sign up my next math homework assignment"

[0547] "Please tell me the details of your homework."

[0548] "Complete the problems on pages 10 to 15. Due next Monday."

[0549] "It's only half done"

[0550] The system automates a series of processes, from voice input to database management, reminder generation, progress tracking and sharing, streamlining the management of submissions.

[0551] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0552] Step 1:

[0553] Dictate submission information

[0554] The user inputs the information for the submission by voice. For example, the user speaks into the microphone of their smartphone or PC, saying, "Register my next math homework." Voice data is acquired as input. The device collects this voice data.

[0555] Specific behavior:

[0556] The user opens the app on their smartphone, presses the "Start Recording" button, and begins speaking. The audio data is input into the device via the microphone.

[0557] Step 2:

[0558] Converting audio data to text

[0559] The device sends the collected voice data to a voice recognition module (for example, Google Cloud Speech-to-Text API) and converts the voice data into text data. Voice data is used as input and text data is generated as output. For example, the voice saying "Please register my next math homework" is converted into text.

[0560] Specific behavior:

[0561] The device sends the voice data to the speech recognition API and displays the returned text. The device prompts again, "Please tell us the details of your homework," and the user responds, "I will complete the problems on pages 10 to 15. The due date is next Monday," and the device converts this into text.

[0562] Step 3:

[0563] Saving submission information

[0564] The terminal sends text data to the server. The server stores the received information in a database (e.g., MySQL) after implementing SQL injection protection measures. Text data is received as input and stored in the database as output.

[0565] Specific behavior:

[0566] The text data "Math homework: solve problems on pages 10 to 15, due date: next Monday" is sent from the terminal to the server, which then stores it in a database.

[0567] Step 4:

[0568] Check the submission deadline

[0569] The server periodically scans the database to check submission deadlines, using the submission information in the database as input and producing a list of submissions that are approaching due dates as output.

[0570] Specific behavior:

[0571] The server checks the database every day at 2:00 PM to find any assignments that are due soon. "I have math homework due next Monday," it says.

[0572] Step 5:

[0573] Reminder generation and notifications

[0574] The server generates reminders for upcoming submissions. For example, the day before a submission is due, it creates a reminder saying, "Tomorrow's math homework is due, how's your progress?" and sends it to the device. The input is a list of upcoming submissions, and the output is a reminder.

[0575] Specific behavior:

[0576] The server generates a reminder "Tomorrow's math homework is due, how's your progress?" and sends it to the device, which displays it to the user as a push notification.

[0577] Step 6:

[0578] Voice input and sending of progress

[0579] The user reports their progress against the reminder by voice, for example, saying "I'm only halfway done." The device collects this voice, converts it into text data using a speech recognition module, and sends it to the server. The voice data of the progress is used as input, and text data is generated as output.

[0580] Specific behavior:

[0581] The user says "I'm only halfway there," and the device converts the speech into text and sends it to the server.

[0582] Step 7:

[0583] Save your progress and set next reminders

[0584] The server saves the progress in a database, for example, recording it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling it to be generated again on the "morning of the submission date". The progress data is received as input, and the next reminder is set as output.

[0585] Specific behavior:

[0586] The server saves "Progress: 50% complete" in the database and sets the next reminder for "the morning of the submission date."

[0587] Step 8:

[0588] Share your progress

[0589] The server analyzes the progress in real time and notifies the sharing target (e.g., teacher or friend). The progress data is used as input and notification data is generated as output.

[0590] Specific behavior:

[0591] The server generates a notification saying "User A is 50% done, deadline is tomorrow" and sends it to the sharing target. The device receives and displays this notification.

[0592] (Application example 1)

[0593] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0594] In factory production line work, workers are required to efficiently manage progress and task status. Conventional methods require a lot of manual input and confirmation work, which not only takes time and effort but also risks progress management errors and delayed reminders. Furthermore, it is difficult to share progress in real time, which can reduce the efficiency of the entire production line. To solve these issues, an efficient and automated progress management system that utilizes voice input is needed.

[0595] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0596] In this invention, the server includes means for inputting information about submissions through voice recognition, means for storing the information about the submissions in a database, means for checking deadlines for submissions and generating reminders, means for notifying users of the reminders, means for users to input progress status through voice recognition, means for saving the progress status in a database, means for notifying sharing targets of the progress status, and means for generating next work instructions based on updates to the progress status. This makes it possible to efficiently manage task progress on production lines in factories and automatically generate and notify reminders. Furthermore, sharing progress status and generating next work instructions in real time can improve the efficiency of the entire production line.

[0597] Below are definitions of important words.

[0598] "Submission information" refers to details of the task or work that a user registers through voice input.

[0599] The "voice recognition means" is a module that has the function of acquiring the words spoken by the user as data and converting the voice data into text data.

[0600] "Means for storing in a database" refers to a device or function that uses a secondary storage device to centrally store information about submissions and progress status.

[0601] The "reminder generator" is a function that automatically creates reminders to notify users at appropriate times based on deadlines for submissions.

[0602] A "means for notifying a reminder" is a device or function that sends the generated reminder to the user via push notification, email, or the like.

[0603] "Progress" refers to information reported by a user about the degree of completion or current progress of a task or submission.

[0604] "Means for inputting progress status by voice recognition" refers to a device or function that allows a user to report progress status by voice and convert it into text data.

[0605] The "means for generating the next work instruction" is a function that automatically determines the next task and work content to be done based on the current progress status and generates instructions.

[0606] The "means for notifying the sharing target of the progress status" is a device or function for reporting the progress status to the sharing target (such as a superior or colleague) in real time.

[0607] This invention is a system for efficiently managing task progress on a production line in a factory. Specifically, it improves production efficiency by registering task details and deadlines through voice input, saving them in a database, and managing the progress in real time.

[0608] System Configuration

[0609] This system mainly consists of a user, a device (such as a smartphone), and a server. The user inputs task details and deadlines by voice via their smartphone, and this data is converted into text data through a voice recognition module. The voice recognition module can be, for example, the SpeechRecognition library.

[0610] Speak and remember submission information

[0611] When a user uses their smartphone to say, "Please add a new task," the device collects voice data through the microphone. At this time, the device uses a voice recognition module to convert the collected voice data into text data. For example, if a user says, "In the next process, assemble 10 parts. The deadline is next Monday," the information is converted into text and stored in a database.

[0612] Submission information stored in a database

[0613] The server receives task information sent from the terminal and stores it in a database after implementing security measures. This system uses SQLite to store information. For example, information such as "Part assembly: Assemble 10 parts, Deadline: Next Monday" is stored.

[0614] Check submission deadlines and generate reminders

[0615] The server periodically scans the database to check deadlines for submissions. When a deadline approaches, it generates a reminder, for example, the day before the deadline. The reminder might say something like, "Tomorrow's task 'Assemble 10 parts' is due, please check your progress." This reminder is sent to the user as a push notification.

[0616] Voice input and sending of progress

[0617] After receiving the reminder, the user can report their progress by voice, for example, by saying "I'm only halfway done," and the device will convert that voice into text data and send it to the server.

[0618] Save your progress and set next reminders

[0619] The server records the progress in a database, saving it as "Progress: 50% complete", and sets the next reminder if necessary, for example, scheduling another reminder to be generated "on the morning of the submission date".

[0620] Share your progress

[0621] The server analyzes the progress in real time and notifies the sharing target (such as a superior or colleague). For example, progress can be shared in the form of "Progress: User A is 50% complete, deadline is tomorrow."

[0622] Specific examples

[0623] For example, worker A might say to his smartphone, "Please add the task of assembling parts," and then voice-input, "Assemble 10 parts. Deadline is next Monday." This information is converted into text data and stored in a database. The day before the deadline, a push notification arrives saying, "Task deadline is approaching," and if worker A reports, "It's only half done," the progress is recorded as 50%. The supervisor can then keep track of the progress in real time.

[0624] Prompt Sentence Examples

[0625] "Please add a task for the next production line: Please tell us the details of the task."

[0626] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0627] Step 1: User dictates task information

[0628] The user uses the smartphone's microphone to input task details and deadlines by voice. For example, when the user says, "Add a new task," the device collects the voice data, and the user utters, "In the next process, assemble 10 parts. The deadline is next Monday." The input data is voice data, which is converted into text data in the next step.

[0629] Step 2: The device converts the audio data into text data.

[0630] A speech recognition module (e.g., SpeechRecognition library) installed on the device acquires voice data and converts it into text data. The input is the user's voice data, and the output is text data. Specifically, the text data obtained is "In the next process, assemble 10 parts. The deadline is next Monday."

[0631] Step 3: The server saves the text data to the database

[0632] The server receives text data sent from the terminal and stores it in a database. The input is text data, and the output is a record stored in the database. The server stores information such as "Part assembly: Assemble 10 parts, Deadline: Next Monday" while implementing security measures.

[0633] Step 4: The server checks the submission deadline and generates a reminder

[0634] The server periodically scans the database to check deadlines for submissions. The input is the deadline information in the database, and the output is a reminder. When the deadline approaches (e.g., the day before the deadline), the server generates a reminder saying, "Tomorrow's task 'Assemble 10 parts' is due, please check your progress."

[0635] Step 5: The device notifies the user of the reminder

[0636] The server sends the generated reminder to the device, and the device notifies the user via a push notification. The input is the generated reminder, and the output is a notification displayed on the user's smartphone. Specifically, a notification appears on the user's screen saying, "Tomorrow's task 'Assemble 10 parts' is due. Please check your progress."

[0637] Step 6: User dictates progress

[0638] After receiving the reminder, the user reports their progress by voice, for example, saying, "I'm only halfway done." The input is the user's voice data, which is converted to text data in the next step.

[0639] Step 7: The device converts the progress voice data into text data.

[0640] The speech recognition module converts the user's voice data into text data. The input is the voice data of the progress, and the output is text data. Specifically, the text data obtained is "It's only half done."

[0641] Step 8: The server saves the progress to a database

[0642] The server receives the text data of the progress status sent from the device and stores it in the database. The input is the text data of the progress status, and the output is a record stored in the database. Specifically, it is stored as "Progress: 50% complete."

[0643] Step 9: The server notifies the sharer of its progress

[0644] The server analyzes the progress in real time and notifies the sharing target (superior or colleague). The input is the progress data, and the output is a notification message. Specifically, it is shared as "Progress content: User A is 50% progress, deadline is tomorrow."

[0645] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0646] The present invention combines an AI system that supports submission management with an emotion engine, and is a system that recognizes the user's emotions and can more effectively manage submissions and notify progress based on those emotions. The following describes specific embodiments of the system.

[0647] System Configuration

[0648] This system allows users to input information about submissions by voice, stores it in a database, generates and notifies reminders at appropriate times, and notifies sharing targets of progress.In addition, it recognizes the user's emotions and adjusts the content of the dialogue and reminders.The system is mainly composed of four parties: the server, the terminal, the user, and the emotion engine.

[0649] Program processing explanation

[0650] Speak and remember submission information

[0651] When a user enters information about their assignment by voice, the device collects the voice data. The device then converts the voice data into text data using a voice recognition module. For example, if the user says, "Register my next math homework," the device responds with, "Please tell me the details of the homework," and if the user says, "Please solve the problems on pages 10 to 15. The due date is next Monday," the device converts the data into text data. This text data is then sent from the device to the server.

[0652] Submission information stored in a database

[0653] The server receives the submitted information from the device, takes measures against SQL injection, and stores it in a database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored.

[0654] Check submission deadlines and generate reminders

[0655] The server periodically scans the database to check when assignments are due. When a deadline approaches, it generates a reminder, for example, the day before. The reminder might be something like, "Your math homework is due tomorrow, how's it going?"

[0656] Reminder notifications

[0657] The server generates a reminder and sends it to the device. The device receives the reminder and displays it to the user as a push notification. This allows the user to receive reminders at the appropriate time without forgetting the submission deadline.

[0658] Voice input and sending of progress

[0659] The user reports their progress against the reminder by voice. For example, if they say, "I'm only halfway done," the device collects the voice and converts it into text data using a speech recognition module. The converted text data becomes, "I'm only halfway done."

[0660] Save your progress and set next reminders

[0661] The server saves the progress to the database, recording it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling another reminder to be generated "on the morning of the submission date".

[0662] Share your progress

[0663] The server analyzes the progress in real time and notifies the sharing target (teacher or friend). For example, it may notify the progress such as "User A is 50% complete, deadline is tomorrow." The device receives this notification and displays it to the sharing target.

[0664] Implementing the Emotion Engine

[0665] The emotion engine is a module that analyzes the user's voice data and determines their emotional state. When a user voice-enters information about a submission or reports progress, the engine analyzes the voice data to recognize the user's emotional state.

[0666] Emotion-aware reminder adjustment

[0667] The server adjusts the content and timing of reminders based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, the server may adjust the content and timing of reminders to be softer or to send them less frequently.

[0668] Customize notifications with emotion recognition

[0669] The server customizes the progress notification content based on the user's emotions recognized by the emotion engine. For example, if the user is feeling impatient, the notification content can be adjusted by adding an encouraging comment.

[0670] Specific examples

[0671] Specifically, a user can voice-input information such as "solve the problems on pages 10 to 15" for math homework, and that information is stored in a database. A reminder is then generated the day before the deadline and sent to the device. If the emotion engine recognizes impatience or anxiety when the user reports "only halfway done," the server adds a comforting message to the next reminder: "Keep up the good work, you're almost there!" The progress is also recognized as 50% complete, and the sharing target is notified of this information.

[0672] In this way, this system combines voice input, database management, reminder functions, progress tracking and sharing functions, as well as an emotion engine to streamline the management of submissions and provide support that is sensitive to the user's emotions.

[0673] The processing flow will be explained below.

[0674] Step 1:

[0675] The user enters the submission information by voice, for example, saying, "AI, register my next math homework."

[0676] Step 2:

[0677] The device collects the voice data and converts it into text data using a voice recognition module. The converted text data becomes "Register my next math homework."

[0678] Step 3:

[0679] The device asks the user to confirm the voice recognition result and ask for additional information. The device asks the user, "Please tell me the details of your homework."

[0680] Step 4:

[0681] The user provides details by voice, for example, "Complete the problems on pages 10 to 15, due next Monday."

[0682] Step 5:

[0683] The device uses a voice recognition module to convert the detailed information into text data, which reads, "Complete the questions on pages 10 to 15. The deadline is next Monday."

[0684] Step 6:

[0685] The terminal parses the converted text data into JSON format or similar and sends it to the server using an HTTP request.

[0686] Step 7:

[0687] The server parses the received data, applies SQL injection protection, and stores it in a database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is saved.

[0688] Step 8:

[0689] The server periodically scans the database to see when submissions are due, and in this case runs a query to extract submissions that are about to expire.

[0690] Step 9:

[0691] The server generates reminders for upcoming submissions, such as "Your math homework is due tomorrow, how's it going?"

[0692] Step 10:

[0693] The server sends the generated reminder to the device, which then notifies the user of the reminder using the Push notification API.

[0694] Step 11:

[0695] The user receives reminders and reports their progress verbally, for example, "I'm only halfway done."

[0696] Step 12:

[0697] The device collects the voice recording of the progress and converts it into text data using the speech recognition module. The converted text data is "It's only half done."

[0698] Step 13:

[0699] The terminal sends the converted text data to the server, and transfers the data to the server using an HTTP request.

[0700] Step 14:

[0701] The server stores the received progress report in a database, for example recording the progress as "50% complete."

[0702] Step 15:

[0703] The server will set the next reminder as needed based on the progress, for example, scheduling another reminder to be generated on the morning of the submission date.

[0704] Step 16:

[0705] The server analyzes the progress in real time and notifies the sharing target. For example, it generates information such as "User A is 50% done, deadline is tomorrow."

[0706] Step 17:

[0707] The server sends progress notifications to the recipients (teachers and friends), who receive the notifications via Webhooks or the notification API.

[0708] Step 18:

[0709] The device will receive the notification and display it on the sharing target. For example, the progress will be displayed on the screen of a smartphone or PC.

[0710] Step 19:

[0711] When a user expresses emotion during a reminder or report, the device sends the voice data to the emotion engine.

[0712] Step 20:

[0713] The emotion engine analyzes the voice data and recognizes the user's emotional state, for example, recognizing that the user is feeling impatient.

[0714] Step 21:

[0715] The server adjusts the content and timing of reminders based on the analysis results of the emotion engine, for example, by making the reminder content softer or reducing the frequency of reminders.

[0716] Step 22:

[0717] The server customizes the progress notification content based on the analysis results of the emotion engine, for example by adding encouraging comments.

[0718] In this way, a system that combines an emotion engine makes it possible to manage submissions in accordance with the user's emotions, thereby providing more personalized support.

[0719] Example 2

[0720] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0721] Conventional submission management systems lack support that takes into account the user's emotional state, which often leads to situations where users feel stressed or anxious. Furthermore, users' progress is rarely shared or notified in real time, making it difficult to respond in a timely manner. This results in inefficient submission management and increases the burden on users.

[0722] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for inputting information about a submission by voice recognition, means for storing information about the submission in a database, means for checking the deadline for the submission and generating a reminder, means for notifying the user of the reminder, means for the user to input progress by voice recognition, means for saving the progress in a database, means for notifying sharing targets of the progress, means for analyzing the user's emotions and adjusting the reminder, and means for analyzing the user's emotions and customizing the notification content. This makes it possible to efficiently manage submissions and provide support that is sensitive to the user's emotions.

[0723] "Submission Information" means detailed information about the work or task that a User is required to submit, including the assignment content, due date, and related materials.

[0724] "Speech recognition" refers to the technology that analyzes a user's voice and converts it into text data, making it possible to use voice as an input method.

[0725] "Database" refers to a collection of electronic records that systematically organizes and manages information for rapid retrieval and access. In this context, it is used to store submission information, progress status, etc.

[0726] A "reminder" is a notification or warning message that prompts a user to take a specific action, helping users to remember deadlines for submissions and other matters.

[0727] "Notification" refers to the process by which a system communicates important information to users based on pre-set timing and conditions. Notifications are given via methods such as push notifications and emails.

[0728] "Progress" refers to the work or task execution status of a user. In this case, it shows how much of a particular task has been completed.

[0729] "SQL injection countermeasures" refers to security measures to prevent unauthorized access to a database, thereby ensuring the safety of the database.

[0730] "Emotion engine" refers to technology that analyzes the user's voice data and recognizes the emotions contained within it, enabling customization based on the user's psychological state.

[0731] "Speech recognition module" refers to a combination of hardware and software for analyzing voice data and converting it into text data. Specifically, this includes the use of APIs.

[0732] A "cron job" is a scheduling technique used to automate periodic tasks within a system, such as scanning a database at regular intervals or generating reminders.

[0733] "Firebase Cloud Messaging" refers to a cloud service for sending notifications to mobile and web applications, enabling real-time push notifications.

[0734] "Microsoft Azure Cognitive Services" refers to a collection of cloud-based artificial intelligence services that provide capabilities such as natural language processing, image recognition, and speech recognition, which are used in this case for sentiment analysis.

[0735] "Emotion API" refers to an interface for detecting and analyzing emotions from a user's voice and images, making it possible to recognize the user's emotional state.

[0736] The present invention combines an AI system that supports submission management with an emotion engine, and is a system that recognizes the user's emotions and can more effectively manage submissions and notify progress based on those emotions. The following describes specific embodiments of the system.

[0737] System Configuration

[0738] This system is mainly composed of four elements: a server, a terminal, a user, and an emotion engine. The system allows users to input information about their submissions by voice, stores it in a database, generates and notifies reminders at appropriate times, and notifies the sharing target of progress. In addition, the system recognizes the user's emotions and adjusts the content of the dialogue and reminders.

[0739] Speak and remember submission information

[0740] When a user voice-inputs information about their assignment, the device collects the voice data. The device uses the Google Cloud Speech-to-Text API as a voice recognition module to convert the voice data into text data. For example, if a user says, "Register my next math homework," the device responds with, "Please tell me the details of the homework," and if the user verbally replies, "I will solve the problems on pages 10 to 15. The due date is next Monday," the content is converted into text data. This text data is sent from the device to the server.

[0741] Submission information stored in a database

[0742] The server receives the submitted information from the device, takes measures against SQL injection, and stores it in a MySQL database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored in the database.

[0743] Check submission deadlines and generate reminders

[0744] The server periodically uses a cron job to scan the database and check deadlines for submissions. When a deadline approaches, it generates a reminder, for example the day before. The reminder might be something like "Your math homework is due tomorrow, how's it going?"

[0745] Reminder notifications

[0746] The server generates a reminder and sends it to the device. The device receives the reminder using Firebase Cloud Messaging and displays it to the user as a push notification. This allows users to receive reminders at the appropriate time without forgetting the submission deadline.

[0747] Speech to text progress

[0748] The user reports their progress against the reminder by voice. For example, if they say, "I'm only halfway done," the device collects the voice and converts it into text data using a speech recognition module. The converted text data is "I'm only halfway done." This text data is then sent back to the server.

[0749] Save your progress and set next reminders

[0750] The server saves the progress to the database, recording it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling another reminder to be generated "on the morning of the submission date".

[0751] Share your progress

[0752] The server analyzes the progress in real time and notifies the sharing target (teacher or friend). For example, it may notify the progress such as "User A is 50% done, deadline is tomorrow." The notification is sent via email or a dedicated application and displayed to the sharing target.

[0753] Implementing the Emotion Engine

[0754] The emotion engine uses the Emotion API of Microsoft Azure Cognitive Services to analyze the user's voice data and determine their emotional state. When a user dictates information for submissions or reports progress, the engine recognizes the user's emotional state by analyzing the voice data.

[0755] Emotion-aware reminder adjustment

[0756] The server adjusts the content and timing of reminders based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, the server may adjust the content and timing of reminders to be softer or to send them less frequently.

[0757] Customize notifications with emotion recognition

[0758] The server customizes the progress notification content based on the user's emotions recognized by the emotion engine. For example, if the user is feeling impatient, the server can adjust the notification content by adding an encouraging comment.

[0759] Specific examples

[0760] Specifically, a user can voice-input information such as "solve the problems on pages 10 to 15" for math homework, and that information is stored in a database. A reminder is then generated the day before the deadline and sent to the device. If the emotion engine recognizes impatience or anxiety when the user reports "only halfway done," the server adds a comforting message to the next reminder: "Keep up the good work, you're almost there!" The progress is also recognized as 50% complete, and the sharing target is notified of this information.

[0761] Examples of prompt statements

[0762] An example of a prompt that can be input to a generative AI model is a user's voice input such as "Please register my next math homework." Another example of a prompt is a user's voice input of their progress, such as "I'm only halfway done."

[0763] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0764] Step 1:

[0765] The user inputs the information about the assignment by voice. For example, when the user says, "Register my next math homework," the voice data is input into the terminal. The terminal collects this voice data.

[0766] Step 2:

[0767] The device converts the collected voice data into text using the Google Cloud Speech-to-Text API, which then provides instructions such as "Register your next math homework assignment."

[0768] Step 3:

[0769] The device sends the text data to the server, which then generates a response message including a follow-up question, such as "Please tell me the details of the next math homework assignment," and sends it back to the device.

[0770] Step 4:

[0771] The device receives the response message from the server and prompts the user by voice, "Please tell me the details of your homework." The user then enters the details by voice, saying, "Please solve the problems on pages 10 to 15. The deadline is next Monday," and the device collects the voice data again.

[0772] Step 5:

[0773] The device converts the collected detailed voice data into text using the Google Cloud Speech-to-Text API, resulting in text such as "Complete the questions on pages 10 to 15. The deadline is next Monday."

[0774] Step 6:

[0775] The device then sends the text data back to the server. The server receives this information, applies SQL injection protection, and stores it in a MySQL database. The stored content is "Math homework: solve problems on pages 10 to 15, due date: next Monday."

[0776] Step 7:

[0777] The server periodically scans the database using a cron job to check deadlines for work, for example, if "Math Homework" is due soon, and generates a reminder if necessary: ​​"Your math homework is due tomorrow, how's it going?"

[0778] Step 8:

[0779] The server generates a reminder and sends it to the device. The device receives the reminder using Firebase Cloud Messaging and sends a push notification to the user saying, "Tomorrow's math homework is due. How's the progress?"

[0780] Step 9:

[0781] The user reports their progress by voice. For example, if they report "I'm only halfway done," the device collects that voice and converts it into text data, "I'm only halfway done," using the Google Cloud Speech-to-Text API.

[0782] Step 10:

[0783] The device sends the text data to the server, which records the progress status in the database as "Progress: 50% complete" and sets the next reminder based on the progress data. For example, it schedules another reminder to be generated on the "morning of the submission date."

[0784] Step 11:

[0785] The server analyzes the progress and notifies the sharing target, such as teachers and friends, of the progress information. For example, it may notify users via email or a dedicated application that "User A is 50% complete, deadline is tomorrow."

[0786] Step 12:

[0787] To recognize the user's emotional state as input, the device uses an emotion engine (Microsoft Azure Cognitive Services' Emotion API) to analyze the user's voice data. Through the analysis, it can determine, for example, whether the user is feeling stressed.

[0788] Step 13:

[0789] The server adjusts the content and timing of reminders based on the emotion recognition results. If the user is feeling stressed, the content of the reminders will be softened or the frequency will be reduced.

[0790] Step 14:

[0791] The server customizes the progress notification content based on the emotion recognition results. For example, if the user is feeling impatient, it adds an encouraging comment such as "Keep up the good work, you're almost there!"

[0792] Examples:

[0793] When a user says, "Sign up my next math homework," the device collects the speech, converts it into text data using a speech recognition module, and sends it to the server. The server responds, and after the user enters details, the server uses an emotion engine to recognize the user's stress and adjusts the content of reminders and notifications.

[0794] (Application example 2)

[0795] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0796] Conventional submission management systems provide reminders and notifications without considering the user's emotional state, resulting in a lack of support appropriate to the user's situation and feelings. Furthermore, they lacked an emotion-based product recommendation function, which meant that the user's shopping experience could not be optimized.

[0797] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting information about a submission using voice recognition, means for storing information about the submission in a database, means for checking the deadline for the submission and generating a reminder, means for notifying the user of the reminder, means for the user to input progress status using voice recognition, means for saving the progress status in a database, means for notifying sharing targets of the progress status, means for recognizing the user's emotions from voice and facial expressions, means for adjusting the content and timing of the reminder based on the emotion, and means for recommending products based on the emotion. This enables submission management that is sensitive to the user's emotions and product recommendations based on emotions.

[0798] A "submittable" is a completed product or something to be prepared related to some task or activity.

[0799] "Speech recognition" is a technology that analyzes speech and converts it into text or commands.

[0800] A "database" is a collection of information that is structured so that the data can be efficiently managed and searched.

[0801] A "reminder" is a message sent to a user to notify them of a specific date, time, or event.

[0802] "Progress" is information that indicates the degree of completion or current status of a specific task or work.

[0803] A "shared party" is a person or organization with whom specific data or information is shared.

[0804] "Emotion recognition" is a technology that identifies a user's emotional state by analyzing their voice and facial expressions.

[0805] "Product recommendation" refers to the selection of products or services suggested for purchase based on the user's preferences and status.

[0806] The present invention combines a system for supporting submission management with an emotion engine, and can provide reminders and product recommendations according to the user's emotional state. An embodiment of the present invention will be described in detail below.

[0807] System Configuration

[0808] This system mainly consists of four components: a server, a terminal, a user, and an emotion engine.

[0809] The server manages submission information, checks deadlines, generates and notifies reminders, manages progress, recognizes emotions, and recommends products.

[0810] The device, typically a smartphone or tablet, accepts voice input, converts the voice data into text, and displays reminders and notifications to the user.

[0811] Users provide submission information and progress to the system through voice input and receive reminders and notifications.

[0812] The emotion engine analyzes voice and facial expression data to recognize the user's emotional state, and customizes reminder content and product recommendations based on this emotion recognition.

[0813] Hardware and Software

[0814] The main hardware and software used in this system are as follows:

[0815] Hardware: Smartphone (microphone, camera), server

[0816] Software: speech recognition modules (e.g., Google Speech Recognition API), emotion recognition systems (e.g., Emotion Recognizer), database management systems (e.g., SQLite), notification systems

[0817] Processing flow

[0818] 1. Voice to text conversion:

[0819] Users can provide information about their submissions and progress by speaking into the device. This voice data is collected through the smartphone's microphone and converted into text data using a voice recognition module.

[0820] 2. Database storage:

[0821] Submission information and progress status converted into text data are sent from the device to a server and securely stored in a database management system (e.g., SQLite).

[0822] 3. Emotion recognition:

[0823] The user's voice and facial expression data are analyzed through an emotion engine to recognize their emotional state, and the emotion recognition results are sent to the server in real time.

[0824] 4. Reminder generation and notifications:

[0825] The server checks the deadline for submissions and generates reminders. The content and timing of the reminders are adjusted based on the emotion recognition results. For example, if the user is feeling stressed, the reminder content will be softened. The reminder is then sent to the user's device via a notification system.

[0826] 5. Progress Management and Sharing:

[0827] The user reports their progress by voice, which is stored in a database. The server analyzes the progress in real time and notifies co-users (e.g., teachers, friends).

[0828] 6. Product recommendation:

[0829] Based on the emotion recognition results, the server recommends appropriate products to the user. For example, if the user is feeling stressed, it recommends relaxation products.

[0830] Specific use cases

[0831] For example, a user can say "I want to buy milk" into their smartphone, and the app will add it to the list. It will also send a reminder one week later. Because the user is feeling stressed, the app will recommend products that will help them relax and display a reminder with an encouraging message.

[0832] Prompt Sentence Examples

[0833] "Create a list of products input by the user's voice and recommend products based on their emotions. Build a system that adjusts the content of reminders and notifies them at the appropriate time based on the emotion recognition results."

[0834] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0835] Step 1:

[0836] The user speaks into the device to input information about their submission and progress. The device collects this voice data and converts it into text data using a voice recognition module (e.g., Google Speech Recognition API). For example, if the user speaks, "I want to buy milk," the voice recognition module generates the text data, "I want to buy milk." Input: Voice data, Output: Text data.

[0837] Step 2:

[0838] The terminal sends the generated text data to the server, which stores the submitted work and progress information in a database management system (e.g., SQLite). Input: Text data, Output: Database storage.

[0839] Step 3:

[0840] The user expresses their emotions for the day through their voice and facial expression on the device. The device collects the voice and facial expression data and analyzes their emotional state using an emotion recognition system (e.g., Emotion Recognizer). For example, if the user's voice sounds tense, the emotion recognition system will recognize it as "stress." Input: Voice and facial expression data, Output: Emotional state data.

[0841] Step 4:

[0842] The server periodically scans the submission deadlines stored in the database. When a deadline approaches, it generates a reminder. It adjusts the content and timing of the reminder based on emotion recognition data. For example, if the user is feeling stressed, it generates a gentle reminder such as "The deadline to buy milk is tomorrow. Please stay calm and proceed." Input: submission data, emotional state data, output: reminder data.

[0843] Step 5:

[0844] The server sends the generated reminder data to the device. The device displays the reminder to the user through the notification system, allowing the user to be notified of deadlines and shopping list items at the appropriate time. Input: Reminder data, Output: Reminder notification.

[0845] Step 6:

[0846] The user reports their progress by voice input. The device collects the voice data and converts it into text data using a voice recognition module, just as in step 1. For example, if the user says, "It's only half done," the text data "It's only half done" is generated. Input: Voice data, Output: Text data.

[0847] Step 7:

[0848] The terminal sends the generated text data of the progress to the server. The server saves the progress in a database and analyzes the progress in real time. Input: Text data of the progress, Output: Database storage and analysis results.

[0849] Step 8:

[0850] The server notifies the sharing target (e.g., teacher, friend) of the results of the progress analysis. For example, if a user's progress is 50% complete, a notification stating "User A is 50% complete, deadline is tomorrow" is generated and sent to the sharing target. Input: Progress analysis data, Output: Sharing notification data.

[0851] Step 9:

[0852] The server recommends products to the user based on the emotion recognition results. For example, if the user is feeling stressed, it will recommend relaxation items. Based on the emotion data, the product recommendation algorithm selects the most suitable product and notifies the user. Input: Emotional state data, product database, Output: Product recommendation data.

[0853] Step 10:

[0854] The server sends the recommended product information to the terminal, which then notifies and displays the product information to the user. This makes it easier for the user to purchase products that match their emotional state. Input: product recommendation data, Output: product recommendation notification.

[0855] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0856] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0857] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0858] [Third embodiment]

[0859] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0860] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0861] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0862] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0863] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0864] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0865] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0866] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0867] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0868] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0869] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0870] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0871] The present invention is an AI system that supports submission management and is designed to enable users to effectively manage submissions and understand progress in a timely manner. The following describes an embodiment of the system.

[0872] System Configuration

[0873] This system automates the entire process from users registering submission information by voice input, storing it in a database, generating and notifying reminders at appropriate times, and notifying those sharing the progress.The system is mainly composed of three parties: a server, a terminal, and a user.

[0874] Program processing explanation

[0875] Speak and remember submission information

[0876] When a user inputs information about their submission by voice, the device collects the voice data. The device then uses a voice recognition module to convert this voice data into text data. For example, if a user says, "Register my next math homework," the device responds, "Please tell me the details of the homework," and if the user says, "Complete the problems on pages 10 to 15. The due date is next Monday," the device converts this into text data. This text data is sent from the device to the server.

[0877] Submission information stored in a database

[0878] The server receives the submitted information from the device, takes measures against SQL injection, and stores it in a database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored.

[0879] Check submission deadlines and generate reminders

[0880] The server periodically scans the database to check when assignments are due. When a deadline approaches, it generates a reminder, for example, the day before. The reminder might be something like, "Your math homework is due tomorrow, how's it going?"

[0881] Reminder notifications

[0882] The server generates a reminder and sends it to the device, which receives it and displays it to the user as a push notification, allowing the user to receive timely reminders without forgetting the deadline.

[0883] Voice input and sending of progress

[0884] When a user reports their progress against a reminder by voice, the device collects the voice and converts it into text data using a speech recognition module. For example, if a user says, "I'm only halfway done," the information is sent to the server as text data.

[0885] Save your progress and set next reminders

[0886] The server saves the progress to the database, records it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling it to be generated again on the "morning of the submission date".

[0887] Share your progress

[0888] The server analyzes the progress in real time and notifies the sharing target (teacher or friend). For example, it may notify the progress such as "User A is 50% complete, deadline is tomorrow." The device receives this notification and displays it to the sharing target.

[0889] Specific examples

[0890] Specifically, the user voice-enters information such as "solve the problems on pages 10 to 15" for math homework, and the information is stored in a database. A reminder is then generated the day before the submission deadline and sent to the device. If the user reports their progress as "only half done," the information is recorded in the database again, and the progress is set to 50% complete. The progress is also notified to the teacher, allowing both the user and the teacher to keep track of the progress of the submission.

[0891] In this way, the system combines voice input, database management, reminder functions, progress tracking and sharing functions to streamline the management of submissions.

[0892] The processing flow will be explained below.

[0893] Step 1:

[0894] The user enters the submission information by voice, for example, saying, "AI, register my next math homework."

[0895] Step 2:

[0896] The device collects the voice data and converts it into text data using a voice recognition module. The converted text data becomes "Register my next math homework."

[0897] Step 3:

[0898] The device asks the user to confirm the voice recognition result and ask for additional information. The device asks the user, "Please tell me the details of your homework."

[0899] Step 4:

[0900] The user provides details by voice, for example, "Complete the problems on pages 10 to 15, due next Monday."

[0901] Step 5:

[0902] The device uses a voice recognition module to convert the detailed information into text data, which reads, "Complete the questions on pages 10 to 15. The deadline is next Monday."

[0903] Step 6:

[0904] The terminal parses the converted text data into JSON format or similar and sends it to the server using an HTTP request.

[0905] Step 7:

[0906] The server parses the received data, applies SQL injection protection, and stores it in a database. For example, "Math homework: solve problems on pages 10 to 15, due date: next Monday" is saved.

[0907] Step 8:

[0908] The server periodically scans the database to see when submissions are due, and in this case runs a query to extract submissions that are about to expire.

[0909] Step 9:

[0910] The server generates reminders for upcoming submissions, such as "Your math homework is due tomorrow, how's it going?"

[0911] Step 10:

[0912] The server sends the generated reminder to the device, which then notifies the user of the reminder using the Push notification API.

[0913] Step 11:

[0914] The user receives reminders and reports their progress verbally, for example, "I'm only halfway done."

[0915] Step 12:

[0916] The device collects the voice recording of the progress and converts it into text data using the speech recognition module. The converted text data is "It's only half done."

[0917] Step 13:

[0918] The terminal sends the progress report text data to the server. The data is transferred to the server using an HTTP request.

[0919] Step 14:

[0920] The server stores the received progress report in a database, for example recording the progress as "50% complete."

[0921] Step 15:

[0922] The server will set the next reminder as needed based on the progress, for example, scheduling another reminder to be generated on the morning of the submission date.

[0923] Step 16:

[0924] The server analyzes the progress in real time and notifies the sharing target. For example, it generates information such as "User A is 50% done, deadline is tomorrow."

[0925] Step 17:

[0926] The server sends progress notifications to the recipients (teachers and friends), who receive the notifications via Webhooks or the notification API.

[0927] Step 18:

[0928] The device will receive the notification and display it on the sharing target. For example, the progress will be displayed on the screen of a smartphone or PC.

[0929] In this way, users can effectively manage the progress of their submissions and use the reminder function, and teachers and friends can also grasp the progress of their submissions in real time.

[0930] Example 1

[0931] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0932] Conventional submission management systems require users to manually enter submission information, which is cumbersome and time-consuming. Also, deadline reminders and progress management are often done manually, which can lead to forgetting. Furthermore, progress is not shared in real time, which means that stakeholders are unable to grasp the latest information, making it difficult to provide appropriate support and advice.

[0933] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0934] In this invention, the server includes means for inputting submission information by voice recognition, means for storing the submission information in a database, means for checking submission deadlines and generating reminders, means for notifying the user of the reminder, means for the user to input progress status by voice recognition, means for saving the progress status in a database, means for notifying sharing targets of the progress status, and means for analyzing the submission information and progress status in real time and setting an appropriate reminder schedule. This allows users to easily manage submissions by voice input, and enables them to remember submission deadlines by receiving reminders and share progress information with relevant parties in real time.

[0935] "Submission information" is detailed data about homework, assignments, documents, etc. that a user must submit.

[0936] "Speech recognition" is a technology that recognizes a user's voice as digital data and converts it into corresponding text data.

[0937] A "database" is a system for efficiently storing, managing, and searching data such as submission information and progress status.

[0938] A "reminder" is a message or alarm that is periodically sent to prompt a user to take a specific action.

[0939] "Progress" is information that indicates the progress and degree of achievement of the submission that the user is working on until completion.

[0940] "Shared with" refers to other users or interested parties (e.g., teachers, friends) who are set up to receive the user's progress information.

[0941] "Real-time analysis" is a technology that analyzes data immediately at the moment it is generated and extracts or processes the necessary information.

[0942] "Setting a schedule" is the act of planning in advance when a specific event or reminder will occur and managing it in the system.

[0943] The present invention is an AI system that supports submission management and is designed to enable users to efficiently manage submissions and keep track of progress in a timely manner. Specific embodiments of this system are described below.

[0944] System Configuration

[0945] This system is primarily composed of three parties: a server, a terminal, and a user, and automates a series of processes including voice input, database management, reminder functions, progress tracking, and sharing functions.

[0946] 1. Voice input of submission information

[0947] The user inputs the information for the assignment by voice. The device (smartphone or PC) collects the voice and converts it into text using a speech recognition module (such as Google Cloud Speech-to-Text API). For example, if the user says, "Register my next math homework," the device responds with, "Please tell me the details of the homework." If the user says, "Complete the problems on pages 10 to 15. The due date is next Monday," the device converts this into text data and sends it to the server.

[0948] 2. Database storage of submission information

[0949] The server receives the submitted information sent from the device, and stores it in a database (e.g., MySQL) after implementing SQL injection protection. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored.

[0950] 3. Check submission deadlines and generate reminders

[0951] The server periodically scans the database for deadlines, and generates reminders as they approach, for example the day before. The reminder might be something like, "Your math homework is due tomorrow, how's it going?"

[0952] 4. Reminder Notifications

[0953] The server generates a reminder and sends it to the device, which receives it and displays it to the user as a push notification, allowing the user to receive timely reminders without forgetting the deadline.

[0954] 5. Voice input and sending of progress

[0955] The user reports their progress against the reminder by voice. For example, they might say, "I'm only halfway done." The device collects the voice and converts it into text using a speech recognition module. The text data is then sent to the server.

[0956] 6. Save your progress and set your next reminder

[0957] The server saves the progress to a database, recording it as "Progress: 50% complete." It then sets the next reminder appropriately, scheduling it to be generated again on the morning of the submission date, for example.

[0958] 7. Share your progress

[0959] The server analyzes the progress in real time and notifies the sharing target (for example, a teacher or friend). For example, information such as "User A is 50% complete, deadline is tomorrow" is notified to the sharing target. The device receives this notification and displays it to the relevant parties.

[0960] Specific examples

[0961] For example, a user might say, "Register my next math homework assignment," and then enter specific details such as, "Complete the problems on pages 10 to 15. Due date: next Monday." This information is stored in the database, and a reminder is generated and sent to the device the day before the deadline. If the user receives the reminder and reports their progress as "only half done," this information is recorded in the database and the progress is marked as 50% complete. This progress status is also notified to the teacher, allowing both the teacher and the user to keep up to date with the latest progress.

[0962] Prompt Sentence Examples

[0963] Here are some example prompts to input to a generative AI model:

[0964] "Sign up my next math homework assignment"

[0965] "Please tell me the details of your homework."

[0966] "Complete the problems on pages 10 to 15. Due next Monday."

[0967] "It's only half done"

[0968] The system automates a series of processes, from voice input to database management, reminder generation, progress tracking and sharing, streamlining the management of submissions.

[0969] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0970] Step 1:

[0971] Dictate submission information

[0972] The user inputs the information for the submission by voice. For example, the user speaks into the microphone of their smartphone or PC, saying, "Register my next math homework." Voice data is acquired as input. The device collects this voice data.

[0973] Specific behavior:

[0974] The user opens the app on their smartphone, presses the "Start Recording" button, and begins speaking. The audio data is input into the device via the microphone.

[0975] Step 2:

[0976] Converting audio data to text

[0977] The device sends the collected voice data to a voice recognition module (for example, Google Cloud Speech-to-Text API) and converts the voice data into text data. Voice data is used as input and text data is generated as output. For example, the voice saying "Please register my next math homework" is converted into text.

[0978] Specific behavior:

[0979] The device sends the voice data to the speech recognition API and displays the returned text. The device prompts again, "Please tell us the details of your homework," and the user responds, "I will complete the problems on pages 10 to 15. The due date is next Monday," and the device converts this into text.

[0980] Step 3:

[0981] Saving submission information

[0982] The terminal sends text data to the server. The server stores the received information in a database (e.g., MySQL) after implementing SQL injection protection measures. Text data is received as input and stored in the database as output.

[0983] Specific behavior:

[0984] The text data "Math homework: solve problems on pages 10 to 15, due date: next Monday" is sent from the terminal to the server, which then stores it in a database.

[0985] Step 4:

[0986] Check the submission deadline

[0987] The server periodically scans the database to check submission deadlines, using the submission information in the database as input and producing a list of submissions that are approaching due dates as output.

[0988] Specific behavior:

[0989] The server checks the database every day at 2:00 PM to find any assignments that are due soon. "I have math homework due next Monday," it says.

[0990] Step 5:

[0991] Reminder generation and notifications

[0992] The server generates reminders for upcoming submissions. For example, the day before a submission is due, it creates a reminder saying, "Tomorrow's math homework is due, how's your progress?" and sends it to the device. The input is a list of upcoming submissions, and the output is a reminder.

[0993] Specific behavior:

[0994] The server generates a reminder "Tomorrow's math homework is due, how's your progress?" and sends it to the device, which displays it to the user as a push notification.

[0995] Step 6:

[0996] Voice input and sending of progress

[0997] The user reports their progress against the reminder by voice, for example, saying "I'm only halfway done." The device collects this voice, converts it into text data using a speech recognition module, and sends it to the server. The voice data of the progress is used as input, and text data is generated as output.

[0998] Specific behavior:

[0999] The user says "I'm only halfway there," and the device converts the speech into text and sends it to the server.

[1000] Step 7:

[1001] Save your progress and set next reminders

[1002] The server saves the progress in a database, for example, recording it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling it to be generated again on the "morning of the submission date". The progress data is received as input, and the next reminder is set as output.

[1003] Specific behavior:

[1004] The server saves "Progress: 50% complete" in the database and sets the next reminder for "the morning of the submission date."

[1005] Step 8:

[1006] Share your progress

[1007] The server analyzes the progress in real time and notifies the sharing target (e.g., teacher or friend). The progress data is used as input and notification data is generated as output.

[1008] Specific behavior:

[1009] The server generates a notification saying "User A is 50% done, deadline is tomorrow" and sends it to the sharing target. The device receives and displays this notification.

[1010] (Application example 1)

[1011] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1012] In factory production line work, workers are required to efficiently manage progress and task status. Conventional methods require a lot of manual input and confirmation work, which not only takes time and effort but also risks progress management errors and delayed reminders. Furthermore, it is difficult to share progress in real time, which can reduce the efficiency of the entire production line. To solve these issues, an efficient and automated progress management system that utilizes voice input is needed.

[1013] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1014] In this invention, the server includes means for inputting information about submissions through voice recognition, means for storing the information about the submissions in a database, means for checking deadlines for submissions and generating reminders, means for notifying users of the reminders, means for users to input progress status through voice recognition, means for saving the progress status in a database, means for notifying sharing targets of the progress status, and means for generating next work instructions based on updates to the progress status. This makes it possible to efficiently manage task progress on production lines in factories and automatically generate and notify reminders. Furthermore, sharing progress status and generating next work instructions in real time can improve the efficiency of the entire production line.

[1015] Below are definitions of important words.

[1016] "Submission information" refers to details of the task or work that a user registers through voice input.

[1017] The "voice recognition means" is a module that has the function of acquiring the words spoken by the user as data and converting the voice data into text data.

[1018] "Means for storing in a database" refers to a device or function that uses a secondary storage device to centrally store information about submissions and progress status.

[1019] The "reminder generator" is a function that automatically creates reminders to notify users at appropriate times based on deadlines for submissions.

[1020] A "means for notifying a reminder" is a device or function that sends the generated reminder to the user via push notification, email, or the like.

[1021] "Progress" refers to information reported by a user about the degree of completion or current progress of a task or submission.

[1022] "Means for inputting progress status by voice recognition" refers to a device or function that allows a user to report progress status by voice and convert it into text data.

[1023] The "means for generating the next work instruction" is a function that automatically determines the next task and work content to be done based on the current progress status and generates instructions.

[1024] The "means for notifying the sharing target of the progress status" is a device or function for reporting the progress status to the sharing target (such as a superior or colleague) in real time.

[1025] This invention is a system for efficiently managing task progress on a production line in a factory. Specifically, it improves production efficiency by registering task details and deadlines through voice input, saving them in a database, and managing the progress in real time.

[1026] System Configuration

[1027] This system mainly consists of a user, a device (such as a smartphone), and a server. The user inputs task details and deadlines by voice via their smartphone, and this data is converted into text data through a voice recognition module. The voice recognition module can be, for example, the SpeechRecognition library.

[1028] Speak and remember submission information

[1029] When a user uses their smartphone to say, "Please add a new task," the device collects voice data through the microphone. At this time, the device uses a voice recognition module to convert the collected voice data into text data. For example, if a user says, "In the next process, assemble 10 parts. The deadline is next Monday," the information is converted into text and stored in a database.

[1030] Submission information stored in a database

[1031] The server receives task information sent from the terminal and stores it in a database after implementing security measures. This system uses SQLite to store information. For example, information such as "Part assembly: Assemble 10 parts, Deadline: Next Monday" is stored.

[1032] Check submission deadlines and generate reminders

[1033] The server periodically scans the database to check deadlines for submissions. When a deadline approaches, it generates a reminder, for example, the day before the deadline. The reminder might say something like, "Tomorrow's task 'Assemble 10 parts' is due, please check your progress." This reminder is sent to the user as a push notification.

[1034] Voice input and sending of progress

[1035] After receiving the reminder, the user can report their progress by voice, for example, by saying "I'm only halfway done," and the device will convert that voice into text data and send it to the server.

[1036] Save your progress and set next reminders

[1037] The server records the progress in a database, saving it as "Progress: 50% complete", and sets the next reminder if necessary, for example, scheduling another reminder to be generated "on the morning of the submission date".

[1038] Share your progress

[1039] The server analyzes the progress in real time and notifies the sharing target (such as a superior or colleague). For example, progress can be shared in the form of "Progress: User A is 50% complete, deadline is tomorrow."

[1040] Specific examples

[1041] For example, worker A might say to his smartphone, "Please add the task of assembling parts," and then voice-input, "Assemble 10 parts. Deadline is next Monday." This information is converted into text data and stored in a database. The day before the deadline, a push notification arrives saying, "Task deadline is approaching," and if worker A reports, "It's only half done," the progress is recorded as 50%. The supervisor can then keep track of the progress in real time.

[1042] Prompt Sentence Examples

[1043] "Please add a task for the next production line: Please tell us the details of the task."

[1044] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1045] Step 1: User dictates task information

[1046] The user uses the smartphone's microphone to input task details and deadlines by voice. For example, when the user says, "Add a new task," the device collects the voice data, and the user utters, "In the next process, assemble 10 parts. The deadline is next Monday." The input data is voice data, which is converted into text data in the next step.

[1047] Step 2: The device converts the audio data into text data.

[1048] A speech recognition module (e.g., SpeechRecognition library) installed on the device acquires voice data and converts it into text data. The input is the user's voice data, and the output is text data. Specifically, the text data obtained is "In the next process, assemble 10 parts. The deadline is next Monday."

[1049] Step 3: The server saves the text data to the database

[1050] The server receives text data sent from the terminal and stores it in a database. The input is text data, and the output is a record stored in the database. The server stores information such as "Part assembly: Assemble 10 parts, Deadline: Next Monday" while implementing security measures.

[1051] Step 4: The server checks the submission deadline and generates a reminder

[1052] The server periodically scans the database to check deadlines for submissions. The input is the deadline information in the database, and the output is a reminder. When the deadline approaches (e.g., the day before the deadline), the server generates a reminder saying, "Tomorrow's task 'Assemble 10 parts' is due, please check your progress."

[1053] Step 5: The device notifies the user of the reminder

[1054] The server sends the generated reminder to the device, and the device notifies the user via a push notification. The input is the generated reminder, and the output is a notification displayed on the user's smartphone. Specifically, a notification appears on the user's screen saying, "Tomorrow's task 'Assemble 10 parts' is due. Please check your progress."

[1055] Step 6: User dictates progress

[1056] After receiving the reminder, the user reports their progress by voice, for example, saying, "I'm only halfway done." The input is the user's voice data, which is converted to text data in the next step.

[1057] Step 7: The device converts the progress voice data into text data.

[1058] The speech recognition module converts the user's voice data into text data. The input is the voice data of the progress, and the output is text data. Specifically, the text data obtained is "It's only half done."

[1059] Step 8: The server saves the progress to a database

[1060] The server receives the text data of the progress status sent from the device and stores it in the database. The input is the text data of the progress status, and the output is a record stored in the database. Specifically, it is stored as "Progress: 50% complete."

[1061] Step 9: The server notifies the sharer of its progress

[1062] The server analyzes the progress in real time and notifies the sharing target (superior or colleague). The input is the progress data, and the output is a notification message. Specifically, it is shared as "Progress content: User A is 50% progress, deadline is tomorrow."

[1063] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1064] The present invention combines an AI system that supports submission management with an emotion engine, and is a system that recognizes the user's emotions and can more effectively manage submissions and notify progress based on those emotions. The following describes specific embodiments of the system.

[1065] System Configuration

[1066] This system allows users to input information about submissions by voice, stores it in a database, generates and notifies reminders at appropriate times, and notifies sharing targets of progress.In addition, it recognizes the user's emotions and adjusts the content of the dialogue and reminders.The system is mainly composed of four parties: the server, the terminal, the user, and the emotion engine.

[1067] Program processing explanation

[1068] Speak and remember submission information

[1069] When a user enters information about their assignment by voice, the device collects the voice data. The device then converts the voice data into text data using a voice recognition module. For example, if the user says, "Register my next math homework," the device responds with, "Please tell me the details of the homework," and if the user says, "Please solve the problems on pages 10 to 15. The due date is next Monday," the device converts the data into text data. This text data is then sent from the device to the server.

[1070] Submission information stored in a database

[1071] The server receives the submitted information from the device, takes measures against SQL injection, and stores it in a database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored.

[1072] Check submission deadlines and generate reminders

[1073] The server periodically scans the database to check when assignments are due. When a deadline approaches, it generates a reminder, for example, the day before. The reminder might be something like, "Your math homework is due tomorrow, how's it going?"

[1074] Reminder notifications

[1075] The server generates a reminder and sends it to the device. The device receives the reminder and displays it to the user as a push notification. This allows the user to receive reminders at the appropriate time without forgetting the submission deadline.

[1076] Voice input and sending of progress

[1077] The user reports their progress against the reminder by voice. For example, if they say, "I'm only halfway done," the device collects the voice and converts it into text data using a speech recognition module. The converted text data becomes, "I'm only halfway done."

[1078] Save your progress and set next reminders

[1079] The server saves the progress to the database, recording it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling another reminder to be generated "on the morning of the submission date".

[1080] Share your progress

[1081] The server analyzes the progress in real time and notifies the sharing target (teacher or friend). For example, it may notify the progress such as "User A is 50% complete, deadline is tomorrow." The device receives this notification and displays it to the sharing target.

[1082] Implementing the Emotion Engine

[1083] The emotion engine is a module that analyzes the user's voice data and determines their emotional state. When a user voice-enters information about a submission or reports progress, the engine analyzes the voice data to recognize the user's emotional state.

[1084] Emotion-aware reminder adjustment

[1085] The server adjusts the content and timing of reminders based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, the server may adjust the content and timing of reminders to be softer or to send them less frequently.

[1086] Customize notifications with emotion recognition

[1087] The server customizes the progress notification content based on the user's emotions recognized by the emotion engine. For example, if the user is feeling impatient, the notification content can be adjusted by adding an encouraging comment.

[1088] Specific examples

[1089] Specifically, a user can voice-input information such as "solve the problems on pages 10 to 15" for math homework, and that information is stored in a database. A reminder is then generated the day before the deadline and sent to the device. If the emotion engine recognizes impatience or anxiety when the user reports "only halfway done," the server adds a comforting message to the next reminder: "Keep up the good work, you're almost there!" The progress is also recognized as 50% complete, and the sharing target is notified of this information.

[1090] In this way, this system combines voice input, database management, reminder functions, progress tracking and sharing functions, as well as an emotion engine to streamline the management of submissions and provide support that is sensitive to the user's emotions.

[1091] The processing flow will be explained below.

[1092] Step 1:

[1093] The user enters the submission information by voice, for example, saying, "AI, register my next math homework."

[1094] Step 2:

[1095] The device collects the voice data and converts it into text data using a voice recognition module. The converted text data becomes "Register my next math homework."

[1096] Step 3:

[1097] The device asks the user to confirm the voice recognition result and ask for additional information. The device asks the user, "Please tell me the details of your homework."

[1098] Step 4:

[1099] The user provides details by voice, for example, "Complete the problems on pages 10 to 15, due next Monday."

[1100] Step 5:

[1101] The device uses a voice recognition module to convert the detailed information into text data, which reads, "Complete the questions on pages 10 to 15. The deadline is next Monday."

[1102] Step 6:

[1103] The terminal parses the converted text data into JSON format or similar and sends it to the server using an HTTP request.

[1104] Step 7:

[1105] The server parses the received data, applies SQL injection protection, and stores it in a database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is saved.

[1106] Step 8:

[1107] The server periodically scans the database to see when submissions are due, and in this case runs a query to extract submissions that are about to expire.

[1108] Step 9:

[1109] The server generates reminders for upcoming submissions, such as "Your math homework is due tomorrow, how's it going?"

[1110] Step 10:

[1111] The server sends the generated reminder to the device, which then notifies the user of the reminder using the Push notification API.

[1112] Step 11:

[1113] The user receives reminders and reports their progress verbally, for example, "I'm only halfway done."

[1114] Step 12:

[1115] The device collects the voice recording of the progress and converts it into text data using the speech recognition module. The converted text data is "It's only half done."

[1116] Step 13:

[1117] The terminal sends the converted text data to the server, and transfers the data to the server using an HTTP request.

[1118] Step 14:

[1119] The server stores the received progress report in a database, for example recording the progress as "50% complete."

[1120] Step 15:

[1121] The server will set the next reminder as needed based on the progress, for example, scheduling another reminder to be generated on the morning of the submission date.

[1122] Step 16:

[1123] The server analyzes the progress in real time and notifies the sharing target. For example, it generates information such as "User A is 50% done, deadline is tomorrow."

[1124] Step 17:

[1125] The server sends progress notifications to the recipients (teachers and friends), who receive the notifications via Webhooks or the notification API.

[1126] Step 18:

[1127] The device will receive the notification and display it on the sharing target. For example, the progress will be displayed on the screen of a smartphone or PC.

[1128] Step 19:

[1129] When a user expresses emotion during a reminder or report, the device sends the voice data to the emotion engine.

[1130] Step 20:

[1131] The emotion engine analyzes the voice data and recognizes the user's emotional state, for example, recognizing that the user is feeling impatient.

[1132] Step 21:

[1133] The server adjusts the content and timing of reminders based on the analysis results of the emotion engine, for example, by making the reminder content softer or reducing the frequency of reminders.

[1134] Step 22:

[1135] The server customizes the progress notification content based on the analysis results of the emotion engine, for example by adding encouraging comments.

[1136] In this way, a system that combines an emotion engine makes it possible to manage submissions in accordance with the user's emotions, thereby providing more personalized support.

[1137] Example 2

[1138] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1139] Conventional submission management systems lack support that takes into account the user's emotional state, which often leads to situations where users feel stressed or anxious. Furthermore, users' progress is rarely shared or notified in real time, making it difficult to respond in a timely manner. This results in inefficient submission management and increases the burden on users.

[1140] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for inputting information about a submission by voice recognition, means for storing information about the submission in a database, means for checking the deadline for the submission and generating a reminder, means for notifying the user of the reminder, means for the user to input progress by voice recognition, means for saving the progress in a database, means for notifying sharing targets of the progress, means for analyzing the user's emotions and adjusting the reminder, and means for analyzing the user's emotions and customizing the notification content. This makes it possible to efficiently manage submissions and provide support that is sensitive to the user's emotions.

[1141] "Submission Information" means detailed information about the work or task that a User is required to submit, including the assignment content, due date, and related materials.

[1142] "Speech recognition" refers to the technology that analyzes a user's voice and converts it into text data, making it possible to use voice as an input method.

[1143] "Database" refers to a collection of electronic records that systematically organizes and manages information for rapid retrieval and access. In this context, it is used to store submission information, progress status, etc.

[1144] A "reminder" is a notification or warning message that prompts a user to take a specific action, helping users to remember deadlines for submissions and other matters.

[1145] "Notification" refers to the process by which a system communicates important information to users based on pre-set timing and conditions. Notifications are given via methods such as push notifications and emails.

[1146] "Progress" refers to the work or task execution status of a user. In this case, it shows how much of a particular task has been completed.

[1147] "SQL injection countermeasures" refers to security measures to prevent unauthorized access to a database, thereby ensuring the safety of the database.

[1148] "Emotion engine" refers to technology that analyzes the user's voice data and recognizes the emotions contained within it, enabling customization based on the user's psychological state.

[1149] "Speech recognition module" refers to a combination of hardware and software for analyzing voice data and converting it into text data. Specifically, this includes the use of APIs.

[1150] A "cron job" is a scheduling technique used to automate periodic tasks within a system, such as scanning a database at regular intervals or generating reminders.

[1151] "Firebase Cloud Messaging" refers to a cloud service for sending notifications to mobile and web applications, enabling real-time push notifications.

[1152] "Microsoft Azure Cognitive Services" refers to a collection of cloud-based artificial intelligence services that provide capabilities such as natural language processing, image recognition, and speech recognition, which are used in this case for sentiment analysis.

[1153] "Emotion API" refers to an interface for detecting and analyzing emotions from a user's voice and images, making it possible to recognize the user's emotional state.

[1154] The present invention combines an AI system that supports submission management with an emotion engine, and is a system that recognizes the user's emotions and can more effectively manage submissions and notify progress based on those emotions. The following describes specific embodiments of the system.

[1155] System Configuration

[1156] This system is mainly composed of four elements: a server, a terminal, a user, and an emotion engine. The system allows users to input information about their submissions by voice, stores it in a database, generates and notifies reminders at appropriate times, and notifies the sharing target of progress. In addition, the system recognizes the user's emotions and adjusts the content of the dialogue and reminders.

[1157] Speak and remember submission information

[1158] When a user voice-inputs information about their assignment, the device collects the voice data. The device uses the Google Cloud Speech-to-Text API as a voice recognition module to convert the voice data into text data. For example, if a user says, "Register my next math homework," the device responds with, "Please tell me the details of the homework," and if the user verbally replies, "I will solve the problems on pages 10 to 15. The due date is next Monday," the content is converted into text data. This text data is sent from the device to the server.

[1159] Submission information stored in a database

[1160] The server receives the submitted information from the device, takes measures against SQL injection, and stores it in a MySQL database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored in the database.

[1161] Check submission deadlines and generate reminders

[1162] The server periodically uses a cron job to scan the database and check deadlines for submissions. When a deadline approaches, it generates a reminder, for example the day before. The reminder might be something like "Your math homework is due tomorrow, how's it going?"

[1163] Reminder notifications

[1164] The server generates a reminder and sends it to the device. The device receives the reminder using Firebase Cloud Messaging and displays it to the user as a push notification. This allows users to receive reminders at the appropriate time without forgetting the submission deadline.

[1165] Speech to text progress

[1166] The user reports their progress against the reminder by voice. For example, if they say, "I'm only halfway done," the device collects the voice and converts it into text data using a speech recognition module. The converted text data is "I'm only halfway done." This text data is then sent back to the server.

[1167] Save your progress and set next reminders

[1168] The server saves the progress to the database, recording it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling another reminder to be generated "on the morning of the submission date".

[1169] Share your progress

[1170] The server analyzes the progress in real time and notifies the sharing target (teacher or friend). For example, it may notify the progress such as "User A is 50% done, deadline is tomorrow." The notification is sent via email or a dedicated application and displayed to the sharing target.

[1171] Implementing the Emotion Engine

[1172] The emotion engine uses the Emotion API of Microsoft Azure Cognitive Services to analyze the user's voice data and determine their emotional state. When a user dictates information for submissions or reports progress, the engine recognizes the user's emotional state by analyzing the voice data.

[1173] Emotion-aware reminder adjustment

[1174] The server adjusts the content and timing of reminders based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, the server may adjust the content and timing of reminders to be softer or to send them less frequently.

[1175] Customize notifications with emotion recognition

[1176] The server customizes the progress notification content based on the user's emotions recognized by the emotion engine. For example, if the user is feeling impatient, the server can adjust the notification content by adding an encouraging comment.

[1177] Specific examples

[1178] Specifically, a user can voice-input information such as "solve the problems on pages 10 to 15" for math homework, and that information is stored in a database. A reminder is then generated the day before the deadline and sent to the device. If the emotion engine recognizes impatience or anxiety when the user reports "only halfway done," the server adds a comforting message to the next reminder: "Keep up the good work, you're almost there!" The progress is also recognized as 50% complete, and the sharing target is notified of this information.

[1179] Examples of prompt statements

[1180] An example of a prompt that can be input to a generative AI model is a user's voice input such as "Please register my next math homework." Another example of a prompt is a user's voice input of their progress, such as "I'm only halfway done."

[1181] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1182] Step 1:

[1183] The user inputs the information about the assignment by voice. For example, when the user says, "Register my next math homework," the voice data is input into the terminal. The terminal collects this voice data.

[1184] Step 2:

[1185] The device converts the collected voice data into text using the Google Cloud Speech-to-Text API, which then provides instructions such as "Register your next math homework assignment."

[1186] Step 3:

[1187] The device sends the text data to the server, which then generates a response message including a follow-up question, such as "Please tell me the details of the next math homework assignment," and sends it back to the device.

[1188] Step 4:

[1189] The device receives the response message from the server and prompts the user by voice, "Please tell me the details of your homework." The user then enters the details by voice, saying, "Please solve the problems on pages 10 to 15. The deadline is next Monday," and the device collects the voice data again.

[1190] Step 5:

[1191] The device converts the collected detailed voice data into text using the Google Cloud Speech-to-Text API, resulting in text such as "Complete the questions on pages 10 to 15. The deadline is next Monday."

[1192] Step 6:

[1193] The device then sends the text data back to the server. The server receives this information, applies SQL injection protection, and stores it in a MySQL database. The stored content is "Math homework: solve problems on pages 10 to 15, due date: next Monday."

[1194] Step 7:

[1195] The server periodically scans the database using a cron job to check deadlines for work, for example, if "Math Homework" is due soon, and generates a reminder if necessary: ​​"Your math homework is due tomorrow, how's it going?"

[1196] Step 8:

[1197] The server generates a reminder and sends it to the device. The device receives the reminder using Firebase Cloud Messaging and sends a push notification to the user saying, "Tomorrow's math homework is due. How's the progress?"

[1198] Step 9:

[1199] The user reports their progress by voice. For example, if they report "I'm only halfway done," the device collects that voice and converts it into text data, "I'm only halfway done," using the Google Cloud Speech-to-Text API.

[1200] Step 10:

[1201] The device sends the text data to the server, which records the progress status in the database as "Progress: 50% complete" and sets the next reminder based on the progress data. For example, it schedules another reminder to be generated on the "morning of the submission date."

[1202] Step 11:

[1203] The server analyzes the progress and notifies the sharing target, such as teachers and friends, of the progress information. For example, it may notify users via email or a dedicated application that "User A is 50% complete, deadline is tomorrow."

[1204] Step 12:

[1205] To recognize the user's emotional state as input, the device uses an emotion engine (Microsoft Azure Cognitive Services' Emotion API) to analyze the user's voice data. Through the analysis, it can determine, for example, whether the user is feeling stressed.

[1206] Step 13:

[1207] The server adjusts the content and timing of reminders based on the emotion recognition results. If the user is feeling stressed, the content of the reminders will be softened or the frequency will be reduced.

[1208] Step 14:

[1209] The server customizes the progress notification content based on the emotion recognition results. For example, if the user is feeling impatient, it adds an encouraging comment such as "Keep up the good work, you're almost there!"

[1210] Examples:

[1211] When a user says, "Sign up my next math homework," the device collects the speech, converts it into text data using a speech recognition module, and sends it to the server. The server responds, and after the user enters details, the server uses an emotion engine to recognize the user's stress and adjusts the content of reminders and notifications.

[1212] (Application example 2)

[1213] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1214] Conventional submission management systems provide reminders and notifications without considering the user's emotional state, resulting in a lack of support appropriate to the user's situation and feelings. Furthermore, they lacked an emotion-based product recommendation function, which meant that the user's shopping experience could not be optimized.

[1215] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting information about a submission using voice recognition, means for storing information about the submission in a database, means for checking the deadline for the submission and generating a reminder, means for notifying the user of the reminder, means for the user to input progress status using voice recognition, means for saving the progress status in a database, means for notifying sharing targets of the progress status, means for recognizing the user's emotions from voice and facial expressions, means for adjusting the content and timing of the reminder based on the emotion, and means for recommending products based on the emotion. This enables submission management that is sensitive to the user's emotions and product recommendations based on emotions.

[1216] A "submittable" is a completed product or something to be prepared related to some task or activity.

[1217] "Speech recognition" is a technology that analyzes speech and converts it into text or commands.

[1218] A "database" is a collection of information that is structured so that the data can be efficiently managed and searched.

[1219] A "reminder" is a message sent to a user to notify them of a specific date, time, or event.

[1220] "Progress" is information that indicates the degree of completion or current status of a specific task or work.

[1221] A "shared party" is a person or organization with whom specific data or information is shared.

[1222] "Emotion recognition" is a technology that identifies a user's emotional state by analyzing their voice and facial expressions.

[1223] "Product recommendation" refers to the selection of products or services suggested for purchase based on the user's preferences and status.

[1224] The present invention combines a system for supporting submission management with an emotion engine, and can provide reminders and product recommendations according to the user's emotional state. An embodiment of the present invention will be described in detail below.

[1225] System Configuration

[1226] This system mainly consists of four components: a server, a terminal, a user, and an emotion engine.

[1227] The server manages submission information, checks deadlines, generates and notifies reminders, manages progress, recognizes emotions, and recommends products.

[1228] The device, typically a smartphone or tablet, accepts voice input, converts the voice data into text, and displays reminders and notifications to the user.

[1229] Users provide submission information and progress to the system through voice input and receive reminders and notifications.

[1230] The emotion engine analyzes voice and facial expression data to recognize the user's emotional state, and customizes reminder content and product recommendations based on this emotion recognition.

[1231] Hardware and Software

[1232] The main hardware and software used in this system are as follows:

[1233] Hardware: Smartphone (microphone, camera), server

[1234] Software: speech recognition modules (e.g., Google Speech Recognition API), emotion recognition systems (e.g., Emotion Recognizer), database management systems (e.g., SQLite), notification systems

[1235] Processing flow

[1236] 1. Voice to text conversion:

[1237] Users can provide information about their submissions and progress by speaking into the device. This voice data is collected through the smartphone's microphone and converted into text data using a voice recognition module.

[1238] 2. Database storage:

[1239] Submission information and progress status converted into text data are sent from the device to a server and securely stored in a database management system (e.g., SQLite).

[1240] 3. Emotion recognition:

[1241] The user's voice and facial expression data are analyzed through an emotion engine to recognize their emotional state, and the emotion recognition results are sent to the server in real time.

[1242] 4. Reminder generation and notifications:

[1243] The server checks the deadline for submissions and generates reminders. The content and timing of the reminders are adjusted based on the emotion recognition results. For example, if the user is feeling stressed, the reminder content will be softened. The reminder is then sent to the user's device via a notification system.

[1244] 5. Progress Management and Sharing:

[1245] The user reports their progress by voice, which is stored in a database. The server analyzes the progress in real time and notifies co-users (e.g., teachers, friends).

[1246] 6. Product recommendation:

[1247] Based on the emotion recognition results, the server recommends appropriate products to the user. For example, if the user is feeling stressed, it recommends relaxation products.

[1248] Specific use cases

[1249] For example, a user can say "I want to buy milk" into their smartphone, and the app will add it to the list. It will also send a reminder one week later. Because the user is feeling stressed, the app will recommend products that will help them relax and display a reminder with an encouraging message.

[1250] Prompt Sentence Examples

[1251] "Create a list of products input by the user's voice and recommend products based on their emotions. Build a system that adjusts the content of reminders and notifies them at the appropriate time based on the emotion recognition results."

[1252] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1253] Step 1:

[1254] The user speaks into the device to input information about their submission and progress. The device collects this voice data and converts it into text data using a voice recognition module (e.g., Google Speech Recognition API). For example, if the user speaks, "I want to buy milk," the voice recognition module generates the text data, "I want to buy milk." Input: Voice data, Output: Text data.

[1255] Step 2:

[1256] The terminal sends the generated text data to the server, which stores the submitted work and progress information in a database management system (e.g., SQLite). Input: Text data, Output: Database storage.

[1257] Step 3:

[1258] The user expresses their emotions for the day through their voice and facial expression on the device. The device collects the voice and facial expression data and analyzes their emotional state using an emotion recognition system (e.g., Emotion Recognizer). For example, if the user's voice sounds tense, the emotion recognition system will recognize it as "stress." Input: Voice and facial expression data, Output: Emotional state data.

[1259] Step 4:

[1260] The server periodically scans the submission deadlines stored in the database. When a deadline approaches, it generates a reminder. It adjusts the content and timing of the reminder based on emotion recognition data. For example, if the user is feeling stressed, it generates a gentle reminder such as "The deadline to buy milk is tomorrow. Please stay calm and proceed." Input: submission data, emotional state data, output: reminder data.

[1261] Step 5:

[1262] The server sends the generated reminder data to the device. The device displays the reminder to the user through the notification system, allowing the user to be notified of deadlines and shopping list items at the appropriate time. Input: Reminder data, Output: Reminder notification.

[1263] Step 6:

[1264] The user reports their progress by voice input. The device collects the voice data and converts it into text data using a voice recognition module, just as in step 1. For example, if the user says, "It's only half done," the text data "It's only half done" is generated. Input: Voice data, Output: Text data.

[1265] Step 7:

[1266] The terminal sends the generated text data of the progress to the server. The server saves the progress in a database and analyzes the progress in real time. Input: Text data of the progress, Output: Database storage and analysis results.

[1267] Step 8:

[1268] The server notifies the sharing target (e.g., teacher, friend) of the results of the progress analysis. For example, if a user's progress is 50% complete, a notification stating "User A is 50% complete, deadline is tomorrow" is generated and sent to the sharing target. Input: Progress analysis data, Output: Sharing notification data.

[1269] Step 9:

[1270] The server recommends products to the user based on the emotion recognition results. For example, if the user is feeling stressed, it will recommend relaxation items. Based on the emotion data, the product recommendation algorithm selects the most suitable product and notifies the user. Input: Emotional state data, product database, Output: Product recommendation data.

[1271] Step 10:

[1272] The server sends the recommended product information to the terminal, which then notifies and displays the product information to the user. This makes it easier for the user to purchase products that match their emotional state. Input: product recommendation data, Output: product recommendation notification.

[1273] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1274] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1275] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1276] [Fourth embodiment]

[1277] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1278] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1279] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1280] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1281] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1282] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1283] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1284] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1285] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1286] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1287] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1288] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1289] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1290] The present invention is an AI system that supports submission management and is designed to enable users to effectively manage submissions and understand progress in a timely manner. The following describes an embodiment of the system.

[1291] System Configuration

[1292] This system automates the entire process from users registering submission information by voice input, storing it in a database, generating and notifying reminders at appropriate times, and notifying those sharing the progress.The system is mainly composed of three parties: a server, a terminal, and a user.

[1293] Program processing explanation

[1294] Speak and remember submission information

[1295] When a user inputs information about their submission by voice, the device collects the voice data. The device then uses a voice recognition module to convert this voice data into text data. For example, if a user says, "Register my next math homework," the device responds, "Please tell me the details of the homework," and if the user says, "Complete the problems on pages 10 to 15. The due date is next Monday," the device converts this into text data. This text data is sent from the device to the server.

[1296] Submission information stored in a database

[1297] The server receives the submitted information from the device, takes measures against SQL injection, and stores it in a database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored.

[1298] Check submission deadlines and generate reminders

[1299] The server periodically scans the database to check when assignments are due. When a deadline approaches, it generates a reminder, for example, the day before. The reminder might be something like, "Your math homework is due tomorrow, how's it going?"

[1300] Reminder notifications

[1301] The server generates a reminder and sends it to the device, which receives it and displays it to the user as a push notification, allowing the user to receive timely reminders without forgetting the deadline.

[1302] Voice input and sending of progress

[1303] When a user reports their progress against a reminder by voice, the device collects the voice and converts it into text data using a speech recognition module. For example, if a user says, "I'm only halfway done," the information is sent to the server as text data.

[1304] Save your progress and set next reminders

[1305] The server saves the progress to the database, records it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling it to be generated again on the "morning of the submission date".

[1306] Share your progress

[1307] The server analyzes the progress in real time and notifies the sharing target (teacher or friend). For example, it may notify the progress such as "User A is 50% complete, deadline is tomorrow." The device receives this notification and displays it to the sharing target.

[1308] Specific examples

[1309] Specifically, the user voice-enters information such as "solve the problems on pages 10 to 15" for math homework, and the information is stored in a database. A reminder is then generated the day before the submission deadline and sent to the device. If the user reports their progress as "only half done," the information is recorded in the database again, and the progress is set to 50% complete. The progress is also notified to the teacher, allowing both the user and the teacher to keep track of the progress of the submission.

[1310] In this way, the system combines voice input, database management, reminder functions, progress tracking and sharing functions to streamline the management of submissions.

[1311] The processing flow will be explained below.

[1312] Step 1:

[1313] The user enters the submission information by voice, for example, saying, "AI, register my next math homework."

[1314] Step 2:

[1315] The device collects the voice data and converts it into text data using a voice recognition module. The converted text data becomes "Register my next math homework."

[1316] Step 3:

[1317] The device asks the user to confirm the voice recognition result and ask for additional information. The device asks the user, "Please tell me the details of your homework."

[1318] Step 4:

[1319] The user provides details by voice, for example, "Complete the problems on pages 10 to 15, due next Monday."

[1320] Step 5:

[1321] The device uses a voice recognition module to convert the detailed information into text data, which reads, "Complete the questions on pages 10 to 15. The deadline is next Monday."

[1322] Step 6:

[1323] The terminal parses the converted text data into JSON format or similar and sends it to the server using an HTTP request.

[1324] Step 7:

[1325] The server parses the received data, applies SQL injection protection, and stores it in a database. For example, "Math homework: solve problems on pages 10 to 15, due date: next Monday" is saved.

[1326] Step 8:

[1327] The server periodically scans the database to see when submissions are due, and in this case runs a query to extract submissions that are about to expire.

[1328] Step 9:

[1329] The server generates reminders for upcoming submissions, such as "Your math homework is due tomorrow, how's it going?"

[1330] Step 10:

[1331] The server sends the generated reminder to the device, which then notifies the user of the reminder using the Push notification API.

[1332] Step 11:

[1333] The user receives reminders and reports their progress verbally, for example, "I'm only halfway done."

[1334] Step 12:

[1335] The device collects the voice recording of the progress and converts it into text data using the speech recognition module. The converted text data is "It's only half done."

[1336] Step 13:

[1337] The terminal sends the progress report text data to the server. The data is transferred to the server using an HTTP request.

[1338] Step 14:

[1339] The server stores the received progress report in a database, for example recording the progress as "50% complete."

[1340] Step 15:

[1341] The server will set the next reminder as needed based on the progress, for example, scheduling another reminder to be generated on the morning of the submission date.

[1342] Step 16:

[1343] The server analyzes the progress in real time and notifies the sharing target. For example, it generates information such as "User A is 50% done, deadline is tomorrow."

[1344] Step 17:

[1345] The server sends progress notifications to the recipients (teachers and friends), who receive the notifications via Webhooks or the notification API.

[1346] Step 18:

[1347] The device will receive the notification and display it on the sharing target. For example, the progress will be displayed on the screen of a smartphone or PC.

[1348] In this way, users can effectively manage the progress of their submissions and use the reminder function, and teachers and friends can also grasp the progress of their submissions in real time.

[1349] Example 1

[1350] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1351] Conventional submission management systems require users to manually enter submission information, which is cumbersome and time-consuming. Also, deadline reminders and progress management are often done manually, which can lead to forgetting. Furthermore, progress is not shared in real time, which means that stakeholders are unable to grasp the latest information, making it difficult to provide appropriate support and advice.

[1352] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1353] In this invention, the server includes means for inputting submission information by voice recognition, means for storing the submission information in a database, means for checking submission deadlines and generating reminders, means for notifying the user of the reminder, means for the user to input progress status by voice recognition, means for saving the progress status in a database, means for notifying sharing targets of the progress status, and means for analyzing the submission information and progress status in real time and setting an appropriate reminder schedule. This allows users to easily manage submissions by voice input, and enables them to remember submission deadlines by receiving reminders and share progress information with relevant parties in real time.

[1354] "Submission information" is detailed data about homework, assignments, documents, etc. that a user must submit.

[1355] "Speech recognition" is a technology that recognizes a user's voice as digital data and converts it into corresponding text data.

[1356] A "database" is a system for efficiently storing, managing, and searching data such as submission information and progress status.

[1357] A "reminder" is a message or alarm that is periodically sent to prompt a user to take a specific action.

[1358] "Progress" is information that indicates the progress and degree of achievement of the submission that the user is working on until completion.

[1359] "Shared with" refers to other users or interested parties (e.g., teachers, friends) who are set up to receive the user's progress information.

[1360] "Real-time analysis" is a technology that analyzes data immediately at the moment it is generated and extracts or processes the necessary information.

[1361] "Setting a schedule" is the act of planning in advance when a specific event or reminder will occur and managing it in the system.

[1362] The present invention is an AI system that supports submission management and is designed to enable users to efficiently manage submissions and keep track of progress in a timely manner. Specific embodiments of this system are described below.

[1363] System Configuration

[1364] This system is primarily composed of three parties: a server, a terminal, and a user, and automates a series of processes including voice input, database management, reminder functions, progress tracking, and sharing functions.

[1365] 1. Voice input of submission information

[1366] The user inputs the information for the assignment by voice. The device (smartphone or PC) collects the voice and converts it into text using a speech recognition module (such as Google Cloud Speech-to-Text API). For example, if the user says, "Register my next math homework," the device responds with, "Please tell me the details of the homework." If the user says, "Complete the problems on pages 10 to 15. The due date is next Monday," the device converts this into text data and sends it to the server.

[1367] 2. Database storage of submission information

[1368] The server receives the submitted information sent from the device, and stores it in a database (e.g., MySQL) after implementing SQL injection protection. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored.

[1369] 3. Check submission deadlines and generate reminders

[1370] The server periodically scans the database for deadlines, and generates reminders as they approach, for example the day before. The reminder might be something like, "Your math homework is due tomorrow, how's it going?"

[1371] 4. Reminder Notifications

[1372] The server generates a reminder and sends it to the device, which receives it and displays it to the user as a push notification, allowing the user to receive timely reminders without forgetting the deadline.

[1373] 5. Voice input and sending of progress

[1374] The user reports their progress against the reminder by voice. For example, they might say, "I'm only halfway done." The device collects the voice and converts it into text using a speech recognition module. The text data is then sent to the server.

[1375] 6. Save your progress and set your next reminder

[1376] The server saves the progress to a database, recording it as "Progress: 50% complete." It then sets the next reminder appropriately, scheduling it to be generated again on the morning of the submission date, for example.

[1377] 7. Share your progress

[1378] The server analyzes the progress in real time and notifies the sharing target (for example, a teacher or friend). For example, information such as "User A is 50% complete, deadline is tomorrow" is notified to the sharing target. The device receives this notification and displays it to the relevant parties.

[1379] Specific examples

[1380] For example, a user might say, "Register my next math homework assignment," and then enter specific details such as, "Complete the problems on pages 10 to 15. Due date: next Monday." This information is stored in the database, and a reminder is generated and sent to the device the day before the deadline. If the user receives the reminder and reports their progress as "only half done," this information is recorded in the database and the progress is marked as 50% complete. This progress status is also notified to the teacher, allowing both the teacher and the user to keep up to date with the latest progress.

[1381] Prompt Sentence Examples

[1382] Here are some example prompts to input to a generative AI model:

[1383] "Sign up my next math homework assignment"

[1384] "Please tell me the details of your homework."

[1385] "Complete the problems on pages 10 to 15. Due next Monday."

[1386] "It's only half done"

[1387] The system automates a series of processes, from voice input to database management, reminder generation, progress tracking and sharing, streamlining the management of submissions.

[1388] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1389] Step 1:

[1390] Dictate submission information

[1391] The user inputs the information for the submission by voice. For example, the user speaks into the microphone of their smartphone or PC, saying, "Register my next math homework." Voice data is acquired as input. The device collects this voice data.

[1392] Specific behavior:

[1393] The user opens the app on their smartphone, presses the "Start Recording" button, and begins speaking. The audio data is input into the device via the microphone.

[1394] Step 2:

[1395] Converting audio data to text

[1396] The device sends the collected voice data to a voice recognition module (for example, Google Cloud Speech-to-Text API) and converts the voice data into text data. Voice data is used as input and text data is generated as output. For example, the voice saying "Please register my next math homework" is converted into text.

[1397] Specific behavior:

[1398] The device sends the voice data to the speech recognition API and displays the returned text. The device prompts again, "Please tell us the details of your homework," and the user responds, "I will complete the problems on pages 10 to 15. The due date is next Monday," and the device converts this into text.

[1399] Step 3:

[1400] Saving submission information

[1401] The terminal sends text data to the server. The server stores the received information in a database (e.g., MySQL) after implementing SQL injection protection measures. Text data is received as input and stored in the database as output.

[1402] Specific behavior:

[1403] The text data "Math homework: solve problems on pages 10 to 15, due date: next Monday" is sent from the terminal to the server, which then stores it in a database.

[1404] Step 4:

[1405] Check the submission deadline

[1406] The server periodically scans the database to check submission deadlines, using the submission information in the database as input and producing a list of submissions that are approaching due dates as output.

[1407] Specific behavior:

[1408] The server checks the database every day at 2:00 PM to find any assignments that are due soon. "I have math homework due next Monday," it says.

[1409] Step 5:

[1410] Reminder generation and notifications

[1411] The server generates reminders for upcoming submissions. For example, the day before a submission is due, it creates a reminder saying, "Tomorrow's math homework is due, how's your progress?" and sends it to the device. The input is a list of upcoming submissions, and the output is a reminder.

[1412] Specific behavior:

[1413] The server generates a reminder "Tomorrow's math homework is due, how's your progress?" and sends it to the device, which displays it to the user as a push notification.

[1414] Step 6:

[1415] Voice input and sending of progress

[1416] The user reports their progress against the reminder by voice, for example, saying "I'm only halfway done." The device collects this voice, converts it into text data using a speech recognition module, and sends it to the server. The voice data of the progress is used as input, and text data is generated as output.

[1417] Specific behavior:

[1418] The user says "I'm only halfway there," and the device converts the speech into text and sends it to the server.

[1419] Step 7:

[1420] Save your progress and set next reminders

[1421] The server saves the progress in a database, for example, recording it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling it to be generated again on the "morning of the submission date". The progress data is received as input, and the next reminder is set as output.

[1422] Specific behavior:

[1423] The server saves "Progress: 50% complete" in the database and sets the next reminder for "the morning of the submission date."

[1424] Step 8:

[1425] Share your progress

[1426] The server analyzes the progress in real time and notifies the sharing target (e.g., teacher or friend). The progress data is used as input and notification data is generated as output.

[1427] Specific behavior:

[1428] The server generates a notification saying "User A is 50% done, deadline is tomorrow" and sends it to the sharing target. The device receives and displays this notification.

[1429] (Application example 1)

[1430] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1431] In factory production line work, workers are required to efficiently manage progress and task status. Conventional methods require a lot of manual input and confirmation work, which not only takes time and effort but also risks progress management errors and delayed reminders. Furthermore, it is difficult to share progress in real time, which can reduce the efficiency of the entire production line. To solve these issues, an efficient and automated progress management system that utilizes voice input is needed.

[1432] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1433] In this invention, the server includes means for inputting information about submissions through voice recognition, means for storing the information about the submissions in a database, means for checking deadlines for submissions and generating reminders, means for notifying users of the reminders, means for users to input progress status through voice recognition, means for saving the progress status in a database, means for notifying sharing targets of the progress status, and means for generating next work instructions based on updates to the progress status. This makes it possible to efficiently manage task progress on production lines in factories and automatically generate and notify reminders. Furthermore, sharing progress status and generating next work instructions in real time can improve the efficiency of the entire production line.

[1434] Below are definitions of important words.

[1435] "Submission information" refers to details of the task or work that a user registers through voice input.

[1436] The "voice recognition means" is a module that has the function of acquiring the words spoken by the user as data and converting the voice data into text data.

[1437] "Means for storing in a database" refers to a device or function that uses a secondary storage device to centrally store information about submissions and progress status.

[1438] The "reminder generator" is a function that automatically creates reminders to notify users at appropriate times based on deadlines for submissions.

[1439] A "means for notifying a reminder" is a device or function that sends the generated reminder to the user via push notification, email, or the like.

[1440] "Progress" refers to information reported by a user about the degree of completion or current progress of a task or submission.

[1441] "Means for inputting progress status by voice recognition" refers to a device or function that allows a user to report progress status by voice and convert it into text data.

[1442] The "means for generating the next work instruction" is a function that automatically determines the next task and work content to be done based on the current progress status and generates instructions.

[1443] The "means for notifying the sharing target of the progress status" is a device or function for reporting the progress status to the sharing target (such as a superior or colleague) in real time.

[1444] This invention is a system for efficiently managing task progress on a production line in a factory. Specifically, it improves production efficiency by registering task details and deadlines through voice input, saving them in a database, and managing the progress in real time.

[1445] System Configuration

[1446] This system mainly consists of a user, a device (such as a smartphone), and a server. The user inputs task details and deadlines by voice via their smartphone, and this data is converted into text data through a voice recognition module. The voice recognition module can be, for example, the SpeechRecognition library.

[1447] Speak and remember submission information

[1448] When a user uses their smartphone to say, "Please add a new task," the device collects voice data through the microphone. At this time, the device uses a voice recognition module to convert the collected voice data into text data. For example, if a user says, "In the next process, assemble 10 parts. The deadline is next Monday," the information is converted into text and stored in a database.

[1449] Submission information stored in a database

[1450] The server receives task information sent from the terminal and stores it in a database after implementing security measures. This system uses SQLite to store information. For example, information such as "Part assembly: Assemble 10 parts, Deadline: Next Monday" is stored.

[1451] Check submission deadlines and generate reminders

[1452] The server periodically scans the database to check deadlines for submissions. When a deadline approaches, it generates a reminder, for example, the day before the deadline. The reminder might say something like, "Tomorrow's task 'Assemble 10 parts' is due, please check your progress." This reminder is sent to the user as a push notification.

[1453] Voice input and sending of progress

[1454] After receiving the reminder, the user can report their progress by voice, for example, by saying "I'm only halfway done," and the device will convert that voice into text data and send it to the server.

[1455] Save your progress and set next reminders

[1456] The server records the progress in a database, saving it as "Progress: 50% complete", and sets the next reminder if necessary, for example, scheduling another reminder to be generated "on the morning of the submission date".

[1457] Share your progress

[1458] The server analyzes the progress in real time and notifies the sharing target (such as a superior or colleague). For example, progress can be shared in the form of "Progress: User A is 50% complete, deadline is tomorrow."

[1459] Specific examples

[1460] For example, worker A might say to his smartphone, "Please add the task of assembling parts," and then voice-input, "Assemble 10 parts. Deadline is next Monday." This information is converted into text data and stored in a database. The day before the deadline, a push notification arrives saying, "Task deadline is approaching," and if worker A reports, "It's only half done," the progress is recorded as 50%. The supervisor can then keep track of the progress in real time.

[1461] Prompt Sentence Examples

[1462] "Please add a task for the next production line: Please tell us the details of the task."

[1463] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1464] Step 1: User dictates task information

[1465] The user uses the smartphone's microphone to input task details and deadlines by voice. For example, when the user says, "Add a new task," the device collects the voice data, and the user utters, "In the next process, assemble 10 parts. The deadline is next Monday." The input data is voice data, which is converted into text data in the next step.

[1466] Step 2: The device converts the audio data into text data.

[1467] A speech recognition module (e.g., SpeechRecognition library) installed on the device acquires voice data and converts it into text data. The input is the user's voice data, and the output is text data. Specifically, the text data obtained is "In the next process, assemble 10 parts. The deadline is next Monday."

[1468] Step 3: The server saves the text data to the database

[1469] The server receives text data sent from the terminal and stores it in a database. The input is text data, and the output is a record stored in the database. The server stores information such as "Part assembly: Assemble 10 parts, Deadline: Next Monday" while implementing security measures.

[1470] Step 4: The server checks the submission deadline and generates a reminder

[1471] The server periodically scans the database to check deadlines for submissions. The input is the deadline information in the database, and the output is a reminder. When the deadline approaches (e.g., the day before the deadline), the server generates a reminder saying, "Tomorrow's task 'Assemble 10 parts' is due, please check your progress."

[1472] Step 5: The device notifies the user of the reminder

[1473] The server sends the generated reminder to the device, and the device notifies the user via a push notification. The input is the generated reminder, and the output is a notification displayed on the user's smartphone. Specifically, a notification appears on the user's screen saying, "Tomorrow's task 'Assemble 10 parts' is due. Please check your progress."

[1474] Step 6: User dictates progress

[1475] After receiving the reminder, the user reports their progress by voice, for example, saying, "I'm only halfway done." The input is the user's voice data, which is converted to text data in the next step.

[1476] Step 7: The device converts the progress voice data into text data.

[1477] The speech recognition module converts the user's voice data into text data. The input is the voice data of the progress, and the output is text data. Specifically, the text data obtained is "It's only half done."

[1478] Step 8: The server saves the progress to a database

[1479] The server receives the text data of the progress status sent from the device and stores it in the database. The input is the text data of the progress status, and the output is a record stored in the database. Specifically, it is stored as "Progress: 50% complete."

[1480] Step 9: The server notifies the sharer of its progress

[1481] The server analyzes the progress in real time and notifies the sharing target (superior or colleague). The input is the progress data, and the output is a notification message. Specifically, it is shared as "Progress content: User A is 50% progress, deadline is tomorrow."

[1482] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1483] The present invention combines an AI system that supports submission management with an emotion engine, and is a system that recognizes the user's emotions and can more effectively manage submissions and notify progress based on those emotions. The following describes specific embodiments of the system.

[1484] System Configuration

[1485] This system allows users to input information about submissions by voice, stores it in a database, generates and notifies reminders at appropriate times, and notifies sharing targets of progress.In addition, it recognizes the user's emotions and adjusts the content of the dialogue and reminders.The system is mainly composed of four parties: the server, the terminal, the user, and the emotion engine.

[1486] Program processing explanation

[1487] Speak and remember submission information

[1488] When a user enters information about their assignment by voice, the device collects the voice data. The device then converts the voice data into text data using a voice recognition module. For example, if the user says, "Register my next math homework," the device responds with, "Please tell me the details of the homework," and if the user says, "Please solve the problems on pages 10 to 15. The due date is next Monday," the device converts the data into text data. This text data is then sent from the device to the server.

[1489] Submission information stored in a database

[1490] The server receives the submitted information from the device, takes measures against SQL injection, and stores it in a database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored.

[1491] Check submission deadlines and generate reminders

[1492] The server periodically scans the database to check when assignments are due. When a deadline approaches, it generates a reminder, for example, the day before. The reminder might be something like, "Your math homework is due tomorrow, how's it going?"

[1493] Reminder notifications

[1494] The server generates a reminder and sends it to the device. The device receives the reminder and displays it to the user as a push notification. This allows the user to receive reminders at the appropriate time without forgetting the submission deadline.

[1495] Voice input and sending of progress

[1496] The user reports their progress against the reminder by voice. For example, if they say, "I'm only halfway done," the device collects the voice and converts it into text data using a speech recognition module. The converted text data becomes, "I'm only halfway done."

[1497] Save your progress and set next reminders

[1498] The server saves the progress to the database, recording it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling another reminder to be generated "on the morning of the submission date".

[1499] Share your progress

[1500] The server analyzes the progress in real time and notifies the sharing target (teacher or friend). For example, it may notify the progress such as "User A is 50% complete, deadline is tomorrow." The device receives this notification and displays it to the sharing target.

[1501] Implementing the Emotion Engine

[1502] The emotion engine is a module that analyzes the user's voice data and determines their emotional state. When a user voice-enters information about a submission or reports progress, the engine analyzes the voice data to recognize the user's emotional state.

[1503] Emotion-aware reminder adjustment

[1504] The server adjusts the content and timing of reminders based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, the server may adjust the content and timing of reminders to be softer or to send them less frequently.

[1505] Customize notifications with emotion recognition

[1506] The server customizes the progress notification content based on the user's emotions recognized by the emotion engine. For example, if the user is feeling impatient, the notification content can be adjusted by adding an encouraging comment.

[1507] Specific examples

[1508] Specifically, a user can voice-input information such as "solve the problems on pages 10 to 15" for math homework, and that information is stored in a database. A reminder is then generated the day before the deadline and sent to the device. If the emotion engine recognizes impatience or anxiety when the user reports "only halfway done," the server adds a comforting message to the next reminder: "Keep up the good work, you're almost there!" The progress is also recognized as 50% complete, and the sharing target is notified of this information.

[1509] In this way, this system combines voice input, database management, reminder functions, progress tracking and sharing functions, as well as an emotion engine to streamline the management of submissions and provide support that is sensitive to the user's emotions.

[1510] The processing flow will be explained below.

[1511] Step 1:

[1512] The user enters the submission information by voice, for example, saying, "AI, register my next math homework."

[1513] Step 2:

[1514] The device collects the voice data and converts it into text data using a voice recognition module. The converted text data becomes "Register my next math homework."

[1515] Step 3:

[1516] The device asks the user to confirm the voice recognition result and ask for additional information. The device asks the user, "Please tell me the details of your homework."

[1517] Step 4:

[1518] The user provides details by voice, for example, "Complete the problems on pages 10 to 15, due next Monday."

[1519] Step 5:

[1520] The device uses a voice recognition module to convert the detailed information into text data, which reads, "Complete the questions on pages 10 to 15. The deadline is next Monday."

[1521] Step 6:

[1522] The terminal parses the converted text data into JSON format or similar and sends it to the server using an HTTP request.

[1523] Step 7:

[1524] The server parses the received data, applies SQL injection protection, and stores it in a database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is saved.

[1525] Step 8:

[1526] The server periodically scans the database to see when submissions are due, and in this case runs a query to extract submissions that are about to expire.

[1527] Step 9:

[1528] The server generates reminders for upcoming submissions, such as "Your math homework is due tomorrow, how's it going?"

[1529] Step 10:

[1530] The server sends the generated reminder to the device, which then notifies the user of the reminder using the Push notification API.

[1531] Step 11:

[1532] The user receives reminders and reports their progress verbally, for example, "I'm only halfway done."

[1533] Step 12:

[1534] The device collects the voice recording of the progress and converts it into text data using the speech recognition module. The converted text data is "It's only half done."

[1535] Step 13:

[1536] The terminal sends the converted text data to the server, and transfers the data to the server using an HTTP request.

[1537] Step 14:

[1538] The server stores the received progress report in a database, for example recording the progress as "50% complete."

[1539] Step 15:

[1540] The server will set the next reminder as needed based on the progress, for example, scheduling another reminder to be generated on the morning of the submission date.

[1541] Step 16:

[1542] The server analyzes the progress in real time and notifies the sharing target. For example, it generates information such as "User A is 50% done, deadline is tomorrow."

[1543] Step 17:

[1544] The server sends progress notifications to the recipients (teachers and friends), who receive the notifications via Webhooks or the notification API.

[1545] Step 18:

[1546] The device will receive the notification and display it on the sharing target. For example, the progress will be displayed on the screen of a smartphone or PC.

[1547] Step 19:

[1548] When a user expresses emotion during a reminder or report, the device sends the voice data to the emotion engine.

[1549] Step 20:

[1550] The emotion engine analyzes the voice data and recognizes the user's emotional state, for example, recognizing that the user is feeling impatient.

[1551] Step 21:

[1552] The server adjusts the content and timing of reminders based on the analysis results of the emotion engine, for example, by making the reminder content softer or reducing the frequency of reminders.

[1553] Step 22:

[1554] The server customizes the progress notification content based on the analysis results of the emotion engine, for example by adding encouraging comments.

[1555] In this way, a system that combines an emotion engine makes it possible to manage submissions in accordance with the user's emotions, thereby providing more personalized support.

[1556] Example 2

[1557] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1558] Conventional submission management systems lack support that takes into account the user's emotional state, which often leads to situations where users feel stressed or anxious. Furthermore, users' progress is rarely shared or notified in real time, making it difficult to respond in a timely manner. This results in inefficient submission management and increases the burden on users.

[1559] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for inputting information about a submission by voice recognition, means for storing information about the submission in a database, means for checking the deadline for the submission and generating a reminder, means for notifying the user of the reminder, means for the user to input progress by voice recognition, means for saving the progress in a database, means for notifying sharing targets of the progress, means for analyzing the user's emotions and adjusting the reminder, and means for analyzing the user's emotions and customizing the notification content. This makes it possible to efficiently manage submissions and provide support that is sensitive to the user's emotions.

[1560] "Submission Information" means detailed information about the work or task that a User is required to submit, including the assignment content, due date, and related materials.

[1561] "Speech recognition" refers to the technology that analyzes a user's voice and converts it into text data, making it possible to use voice as an input method.

[1562] "Database" refers to a collection of electronic records that systematically organizes and manages information for rapid retrieval and access. In this context, it is used to store submission information, progress status, etc.

[1563] A "reminder" is a notification or warning message that prompts a user to take a specific action, helping users to remember deadlines for submissions and other matters.

[1564] "Notification" refers to the process by which a system communicates important information to users based on pre-set timing and conditions. Notifications are given via methods such as push notifications and emails.

[1565] "Progress" refers to the work or task execution status of a user. In this case, it shows how much of a particular task has been completed.

[1566] "SQL injection countermeasures" refers to security measures to prevent unauthorized access to a database, thereby ensuring the safety of the database.

[1567] "Emotion engine" refers to technology that analyzes the user's voice data and recognizes the emotions contained within it, enabling customization based on the user's psychological state.

[1568] "Speech recognition module" refers to a combination of hardware and software for analyzing voice data and converting it into text data. Specifically, this includes the use of APIs.

[1569] A "cron job" is a scheduling technique used to automate periodic tasks within a system, such as scanning a database at regular intervals or generating reminders.

[1570] "Firebase Cloud Messaging" refers to a cloud service for sending notifications to mobile and web applications, enabling real-time push notifications.

[1571] "Microsoft Azure Cognitive Services" refers to a collection of cloud-based artificial intelligence services that provide capabilities such as natural language processing, image recognition, and speech recognition, which are used in this case for sentiment analysis.

[1572] "Emotion API" refers to an interface for detecting and analyzing emotions from a user's voice and images, making it possible to recognize the user's emotional state.

[1573] The present invention combines an AI system that supports submission management with an emotion engine, and is a system that recognizes the user's emotions and can more effectively manage submissions and notify progress based on those emotions. The following describes specific embodiments of the system.

[1574] System Configuration

[1575] This system is mainly composed of four elements: a server, a terminal, a user, and an emotion engine. The system allows users to input information about their submissions by voice, stores it in a database, generates and notifies reminders at appropriate times, and notifies the sharing target of progress. In addition, the system recognizes the user's emotions and adjusts the content of the dialogue and reminders.

[1576] Speak and remember submission information

[1577] When a user voice-inputs information about their assignment, the device collects the voice data. The device uses the Google Cloud Speech-to-Text API as a voice recognition module to convert the voice data into text data. For example, if a user says, "Register my next math homework," the device responds with, "Please tell me the details of the homework," and if the user verbally replies, "I will solve the problems on pages 10 to 15. The due date is next Monday," the content is converted into text data. This text data is sent from the device to the server.

[1578] Submission information stored in a database

[1579] The server receives the submitted information from the device, takes measures against SQL injection, and stores it in a MySQL database. For example, information such as "Math homework: solve problems on pages 10 to 15, due date: next Monday" is stored in the database.

[1580] Check submission deadlines and generate reminders

[1581] The server periodically uses a cron job to scan the database and check deadlines for submissions. When a deadline approaches, it generates a reminder, for example the day before. The reminder might be something like "Your math homework is due tomorrow, how's it going?"

[1582] Reminder notifications

[1583] The server generates a reminder and sends it to the device. The device receives the reminder using Firebase Cloud Messaging and displays it to the user as a push notification. This allows users to receive reminders at the appropriate time without forgetting the submission deadline.

[1584] Speech to text progress

[1585] The user reports their progress against the reminder by voice. For example, if they say, "I'm only halfway done," the device collects the voice and converts it into text data using a speech recognition module. The converted text data is "I'm only halfway done." This text data is then sent back to the server.

[1586] Save your progress and set next reminders

[1587] The server saves the progress to the database, recording it as "Progress: 50% complete", and sets the next reminder appropriately, for example, scheduling another reminder to be generated "on the morning of the submission date".

[1588] Share your progress

[1589] The server analyzes the progress in real time and notifies the sharing target (teacher or friend). For example, it may notify the progress such as "User A is 50% done, deadline is tomorrow." The notification is sent via email or a dedicated application and displayed to the sharing target.

[1590] Implementing the Emotion Engine

[1591] The emotion engine uses the Emotion API of Microsoft Azure Cognitive Services to analyze the user's voice data and determine their emotional state. When a user dictates information for submissions or reports progress, the engine recognizes the user's emotional state by analyzing the voice data.

[1592] Emotion-aware reminder adjustment

[1593] The server adjusts the content and timing of reminders based on the user's emotions recognized by the emotion engine. For example, if the user is feeling stressed, the server may adjust the content and timing of reminders to be softer or to send them less frequently.

[1594] Customize notifications with emotion recognition

[1595] The server customizes the progress notification content based on the user's emotions recognized by the emotion engine. For example, if the user is feeling impatient, the server can adjust the notification content by adding an encouraging comment.

[1596] Specific examples

[1597] Specifically, a user can voice-input information such as "solve the problems on pages 10 to 15" for math homework, and that information is stored in a database. A reminder is then generated the day before the deadline and sent to the device. If the emotion engine recognizes impatience or anxiety when the user reports "only halfway done," the server adds a comforting message to the next reminder: "Keep up the good work, you're almost there!" The progress is also recognized as 50% complete, and the sharing target is notified of this information.

[1598] Examples of prompt statements

[1599] An example of a prompt that can be input to a generative AI model is a user's voice input such as "Please register my next math homework." Another example of a prompt is a user's voice input of their progress, such as "I'm only halfway done."

[1600] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1601] Step 1:

[1602] The user inputs the information about the assignment by voice. For example, when the user says, "Register my next math homework," the voice data is input into the terminal. The terminal collects this voice data.

[1603] Step 2:

[1604] The device converts the collected voice data into text using the Google Cloud Speech-to-Text API, which then provides instructions such as "Register your next math homework assignment."

[1605] Step 3:

[1606] The device sends the text data to the server, which then generates a response message including a follow-up question, such as "Please tell me the details of the next math homework assignment," and sends it back to the device.

[1607] Step 4:

[1608] The device receives the response message from the server and prompts the user by voice, "Please tell me the details of your homework." The user then enters the details by voice, saying, "Please solve the problems on pages 10 to 15. The deadline is next Monday," and the device collects the voice data again.

[1609] Step 5:

[1610] The device converts the collected detailed voice data into text using the Google Cloud Speech-to-Text API, resulting in text such as "Complete the questions on pages 10 to 15. The deadline is next Monday."

[1611] Step 6:

[1612] The device then sends the text data back to the server. The server receives this information, applies SQL injection protection, and stores it in a MySQL database. The stored content is "Math homework: solve problems on pages 10 to 15, due date: next Monday."

[1613] Step 7:

[1614] The server periodically scans the database using a cron job to check deadlines for work, for example, if "Math Homework" is due soon, and generates a reminder if necessary: ​​"Your math homework is due tomorrow, how's it going?"

[1615] Step 8:

[1616] The server generates a reminder and sends it to the device. The device receives the reminder using Firebase Cloud Messaging and sends a push notification to the user saying, "Tomorrow's math homework is due. How's the progress?"

[1617] Step 9:

[1618] The user reports their progress by voice. For example, if they report "I'm only halfway done," the device collects that voice and converts it into text data, "I'm only halfway done," using the Google Cloud Speech-to-Text API.

[1619] Step 10:

[1620] The device sends the text data to the server, which records the progress status in the database as "Progress: 50% complete" and sets the next reminder based on the progress data. For example, it schedules another reminder to be generated on the "morning of the submission date."

[1621] Step 11:

[1622] The server analyzes the progress and notifies the sharing target, such as teachers and friends, of the progress information. For example, it may notify users via email or a dedicated application that "User A is 50% complete, deadline is tomorrow."

[1623] Step 12:

[1624] To recognize the user's emotional state as input, the device uses an emotion engine (Microsoft Azure Cognitive Services' Emotion API) to analyze the user's voice data. Through the analysis, it can determine, for example, whether the user is feeling stressed.

[1625] Step 13:

[1626] The server adjusts the content and timing of reminders based on the emotion recognition results. If the user is feeling stressed, the content of the reminders will be softened or the frequency will be reduced.

[1627] Step 14:

[1628] The server customizes the progress notification content based on the emotion recognition results. For example, if the user is feeling impatient, it adds an encouraging comment such as "Keep up the good work, you're almost there!"

[1629] Examples:

[1630] When a user says, "Sign up my next math homework," the device collects the speech, converts it into text data using a speech recognition module, and sends it to the server. The server responds, and after the user enters details, the server uses an emotion engine to recognize the user's stress and adjusts the content of reminders and notifications.

[1631] (Application example 2)

[1632] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1633] Conventional submission management systems provide reminders and notifications without considering the user's emotional state, resulting in a lack of support appropriate to the user's situation and feelings. Furthermore, they lacked an emotion-based product recommendation function, which meant that the user's shopping experience could not be optimized.

[1634] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for inputting information about a submission using voice recognition, means for storing information about the submission in a database, means for checking the deadline for the submission and generating a reminder, means for notifying the user of the reminder, means for the user to input progress status using voice recognition, means for saving the progress status in a database, means for notifying sharing targets of the progress status, means for recognizing the user's emotions from voice and facial expressions, means for adjusting the content and timing of the reminder based on the emotion, and means for recommending products based on the emotion. This enables submission management that is sensitive to the user's emotions and product recommendations based on emotions.

[1635] A "submittable" is a completed product or something to be prepared related to some task or activity.

[1636] "Speech recognition" is a technology that analyzes speech and converts it into text or commands.

[1637] A "database" is a collection of information that is structured so that the data can be efficiently managed and searched.

[1638] A "reminder" is a message sent to a user to notify them of a specific date, time, or event.

[1639] "Progress" is information that indicates the degree of completion or current status of a specific task or work.

[1640] A "shared party" is a person or organization with whom specific data or information is shared.

[1641] "Emotion recognition" is a technology that identifies a user's emotional state by analyzing their voice and facial expressions.

[1642] "Product recommendation" refers to the selection of products or services suggested for purchase based on the user's preferences and status.

[1643] The present invention combines a system for supporting submission management with an emotion engine, and can provide reminders and product recommendations according to the user's emotional state. An embodiment of the present invention will be described in detail below.

[1644] System Configuration

[1645] This system mainly consists of four components: a server, a terminal, a user, and an emotion engine.

[1646] The server manages submission information, checks deadlines, generates and notifies reminders, manages progress, recognizes emotions, and recommends products.

[1647] The device, typically a smartphone or tablet, accepts voice input, converts the voice data into text, and displays reminders and notifications to the user.

[1648] Users provide submission information and progress to the system through voice input and receive reminders and notifications.

[1649] The emotion engine analyzes voice and facial expression data to recognize the user's emotional state, and customizes reminder content and product recommendations based on this emotion recognition.

[1650] Hardware and Software

[1651] The main hardware and software used in this system are as follows:

[1652] Hardware: Smartphone (microphone, camera), server

[1653] Software: speech recognition modules (e.g., Google Speech Recognition API), emotion recognition systems (e.g., Emotion Recognizer), database management systems (e.g., SQLite), notification systems

[1654] Processing flow

[1655] 1. Voice to text conversion:

[1656] Users can provide information about their submissions and progress by speaking into the device. This voice data is collected through the smartphone's microphone and converted into text data using a voice recognition module.

[1657] 2. Database storage:

[1658] Submission information and progress status converted into text data are sent from the device to a server and securely stored in a database management system (e.g., SQLite).

[1659] 3. Emotion recognition:

[1660] The user's voice and facial expression data are analyzed through an emotion engine to recognize their emotional state, and the emotion recognition results are sent to the server in real time.

[1661] 4. Reminder generation and notifications:

[1662] The server checks the deadline for submissions and generates reminders. The content and timing of the reminders are adjusted based on the emotion recognition results. For example, if the user is feeling stressed, the reminder content will be softened. The reminder is then sent to the user's device via a notification system.

[1663] 5. Progress Management and Sharing:

[1664] The user reports their progress by voice, which is stored in a database. The server analyzes the progress in real time and notifies co-users (e.g., teachers, friends).

[1665] 6. Product recommendation:

[1666] Based on the emotion recognition results, the server recommends appropriate products to the user. For example, if the user is feeling stressed, it recommends relaxation products.

[1667] Specific use cases

[1668] For example, a user can say "I want to buy milk" into their smartphone, and the app will add it to the list. It will also send a reminder one week later. Because the user is feeling stressed, the app will recommend products that will help them relax and display a reminder with an encouraging message.

[1669] Prompt Sentence Examples

[1670] "Create a list of products input by the user's voice and recommend products based on their emotions. Build a system that adjusts the content of reminders and notifies them at the appropriate time based on the emotion recognition results."

[1671] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1672] Step 1:

[1673] The user speaks into the device to input information about their submission and progress. The device collects this voice data and converts it into text data using a voice recognition module (e.g., Google Speech Recognition API). For example, if the user speaks, "I want to buy milk," the voice recognition module generates the text data, "I want to buy milk." Input: Voice data, Output: Text data.

[1674] Step 2:

[1675] The terminal sends the generated text data to the server, which stores the submitted work and progress information in a database management system (e.g., SQLite). Input: Text data, Output: Database storage.

[1676] Step 3:

[1677] The user expresses their emotions for the day through their voice and facial expression on the device. The device collects the voice and facial expression data and analyzes their emotional state using an emotion recognition system (e.g., Emotion Recognizer). For example, if the user's voice sounds tense, the emotion recognition system will recognize it as "stress." Input: Voice and facial expression data, Output: Emotional state data.

[1678] Step 4:

[1679] The server periodically scans the submission deadlines stored in the database. When a deadline approaches, it generates a reminder. It adjusts the content and timing of the reminder based on emotion recognition data. For example, if the user is feeling stressed, it generates a gentle reminder such as "The deadline to buy milk is tomorrow. Please stay calm and proceed." Input: submission data, emotional state data, output: reminder data.

[1680] Step 5:

[1681] The server sends the generated reminder data to the device. The device displays the reminder to the user through the notification system, allowing the user to be notified of deadlines and shopping list items at the appropriate time. Input: Reminder data, Output: Reminder notification.

[1682] Step 6:

[1683] The user reports their progress by voice input. The device collects the voice data and converts it into text data using a voice recognition module, just as in step 1. For example, if the user says, "It's only half done," the text data "It's only half done" is generated. Input: Voice data, Output: Text data.

[1684] Step 7:

[1685] The terminal sends the generated text data of the progress to the server. The server saves the progress in a database and analyzes the progress in real time. Input: Text data of the progress, Output: Database storage and analysis results.

[1686] Step 8:

[1687] The server notifies the sharing target (e.g., teacher, friend) of the results of the progress analysis. For example, if a user's progress is 50% complete, a notification stating "User A is 50% complete, deadline is tomorrow" is generated and sent to the sharing target. Input: Progress analysis data, Output: Sharing notification data.

[1688] Step 9:

[1689] The server recommends products to the user based on the emotion recognition results. For example, if the user is feeling stressed, it will recommend relaxation items. Based on the emotion data, the product recommendation algorithm selects the most suitable product and notifies the user. Input: Emotional state data, product database, Output: Product recommendation data.

[1690] Step 10:

[1691] The server sends the recommended product information to the terminal, which then notifies and displays the product information to the user. This makes it easier for the user to purchase products that match their emotional state. Input: product recommendation data, Output: product recommendation notification.

[1692] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1693] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1694] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1695] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1696] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1697] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1698] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1699] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1700] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1701] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1702] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1703] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1704] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1705] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1706] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1707] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1708] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1709] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1710] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1711] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1712] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1713] The following is further disclosed regarding the above embodiment.

[1714] (Claim 1)

[1715] means for inputting submission information by voice recognition;

[1716] means for storing said submission information in a database;

[1717] A means to see submission deadlines and generate reminders;

[1718] a means for notifying the user of the reminder;

[1719] a means for allowing a user to input progress information by voice recognition;

[1720] A means for storing the progress status in a database;

[1721] A means of notifying the sharing target of the progress;

[1722] A system including:

[1723] (Claim 2)

[1724] 10. The system of claim 1, further comprising a speech recognition module for converting information in the submission into text data.

[1725] (Claim 3)

[1726] 10. The system of claim 1, further comprising means for analyzing the progress stored in the database in real time.

[1727] "Example 1"

[1728] (Claim 1)

[1729] means for inputting submission information by voice recognition;

[1730] means for storing said submission information in a database;

[1731] A means to see submission deadlines and generate reminders;

[1732] a means for notifying the user of the reminder;

[1733] a means for allowing a user to input progress information by voice recognition;

[1734] A means for storing the progress status in a database;

[1735] A means of notifying the sharing target of the progress;

[1736] A means to analyze submission information and progress in real time and schedule appropriate reminders;

[1737] A system including:

[1738] (Claim 2)

[1739] 10. The system of claim 1, further comprising a speech recognition module for converting information in the submission into text data.

[1740] (Claim 3)

[1741] 10. The system of claim 1, further comprising means for analyzing the progress stored in the database in real time and notifying the sharing target.

[1742] "Application Example 1"

[1743] (Claim 1)

[1744] means for inputting submission information by voice recognition;

[1745] means for storing said submission information in a database;

[1746] A means to see submission deadlines and generate reminders;

[1747] a means for notifying the user of the reminder;

[1748] a means for allowing a user to input progress information by voice recognition;

[1749] A means for storing the progress status in a database;

[1750] A means of notifying the sharing target of the progress;

[1751] means for generating next work orders based on progress updates;

[1752] A system including:

[1753] (Claim 2)

[1754] 10. The system of claim 1, further comprising a speech recognition module for converting information in the submission into text data.

[1755] (Claim 3)

[1756] 10. The system of claim 1, further comprising means for analyzing the progress stored in the database in real time.

[1757] (Claim 4)

[1758] 2. The system according to claim 1, further comprising a means for notifying a delivery item of its due date the day before based on progress as a function for generating a reminder.

[1759] "Example 2: Combining Emotion Engines"

[1760] (Claim 1)

[1761] means for inputting submission information by voice recognition;

[1762] means for storing said submission information in a database;

[1763] A means to see submission deadlines and generate reminders;

[1764] a means for notifying the user of the reminder;

[1765] a means for allowing a user to input progress information by voice recognition;

[1766] A means for storing the progress status in a database;

[1767] A means of notifying the sharing target of the progress;

[1768] A means of analyzing user sentiment and adjusting reminders;

[1769] A means to analyze user sentiment and customize notification content;

[1770] A system including:

[1771] (Claim 2)

[1772] 10. The system of claim 1, further comp...

Claims

1. means for inputting submission information by voice recognition; means for storing said submission information in a database; A means to see submission deadlines and generate reminders; a means for notifying the user of the reminder; a means for allowing a user to input progress information by voice recognition; A means for storing the progress status in a database; A means of notifying the sharing target of the progress; A system including:

2. The system of claim 1 , further comprising a speech recognition module for converting information in the submission into text data.

3. 10. The system of claim 1, further comprising means for analyzing the progress stored in the database in real time.

Images