System

Abstract

A system is provided.SOLUTION: A system comprising: means for a user to input goals and challenges and upload text and media data; means for receiving and storing the inputted data; means for analyzing the received data and generating feedback for user goal achievement and challenge resolution; and means for providing the generated feedback to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Business people face a lot of work and pressure every day, making it difficult to objectively evaluate and improve their careers and performance. In such situations, hiring a professional human coach is expensive and often lacks a guarantee of reliability. It is also difficult to receive effective coaching because it is unclear which company or personal coach to contact to find a coach who is right for you. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for users to input goals and challenges and upload text data and media data, a means for receiving and storing the data, a means for analyzing the received data and generating feedback for the user to achieve their goals and solve their challenges, and a means for providing the generated feedback to the user. The system also includes a means for continuously tracking the user's progress, providing updated feedback based on the tracking results, and personalizing the feedback content and communication style based on the user's personality and preferences. This allows users to receive highly efficient, personalized coaching at low cost. Furthermore, by using multimodal AI to analyze not only text data but also image, audio, and video data, more effective feedback can be provided.

[0006] "Users" refer to business people who use this system to input their own goals and challenges and receive feedback.

[0007] A "goal" refers to a specific outcome or product that a user is trying to achieve.

[0008] "Issues" refer to problems or areas for improvement that users need to solve.

[0009] "Text data" refers to textual information entered by users, including descriptions of goals and tasks, personal profile information, etc.

[0010] "Media data" refers to multimedia information such as images, audio, and video uploaded by users.

[0011] "Terminal" means the device used by a User to access the System and input / upload Data, including a PC or smartphone.

[0012] "Server" refers to a central computer that receives and stores input data, analyzes it, and generates feedback.

[0013] "Receiving" refers to the process by which the server retrieves data sent from the terminal.

[0014] "Storage" refers to the process of storing received data in a database.

[0015] "Analysis" refers to the process by which the server handles text and media data, understands and processes it to help users achieve their goals and solve their problems.

[0016] "Feedback" refers to advice, opinions, and specific methods of improvement provided to users based on the analysis results.

[0017] "Tracking" refers to the process of continuously recording and observing a user's progress.

[0018] "Personalization" refers to the process of tailoring feedback and communication style to suit the user's personality and preferences.

[0019] "Multimodal AI" refers to artificial intelligence that can analyze not only text data, but also image, audio, and video data. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention is a system that provides effective feedback and coaching to business people to help them achieve their goals and challenges. Specific embodiments for implementing the system will be described below.

[0042] System configuration

[0043] The system mainly includes the following components:

[0044] 1. User's device

[0045] A device (such as a computer or smartphone) on which users can enter their goals and challenges and upload text and media data.

[0046] 2. Server

[0047] A central system that receives, stores, analyzes input data, and provides feedback to users.

[0048] Program processing overview

[0049] 1. Entering user information and initial settings

[0050] A user uses a terminal to input their basic information (such as name, occupation, goals, and tasks). This input includes not only text data, but also image, audio, and video data as needed. For example, a user aiming to succeed in a new project may set improving their presentation skills as a task and upload videos of past presentations to the system.

[0051] 2. Sending and Receiving Data

[0052] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[0053] 3. Data Analysis

[0054] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies, including natural language processing (NLP) of text data, as well as image recognition, speech analysis, and video analysis. For example, a presentation video can be analyzed to generate feedback on pronunciation and the flow of the presentation.

[0055] 4. Create a coaching plan

[0056] Based on the analysis results, the server generates a specific coaching plan for the user, which includes feedback, advice, and a concrete action plan. For example, a training plan for improving presentation skills could be created, with specific points to improve and practice methods.

[0057] 5. Providing Feedback

[0058] The server then sends the generated feedback and advice to the terminal and provides it to the user. The feedback is provided in the form of text, audio, video, etc., and the user can check and use it as appropriate.

[0059] 6. Track your progress

[0060] The server periodically tracks the user's progress: the user periodically uploads new data (e.g., a new presentation video), which the server analyzes and provides up-to-date feedback.

[0061] 7. Personalization and Adaptation

[0062] The server personalizes the feedback and communication style based on the user's personality and preferences, resulting in more effective coaching. For example, it adopts a positive feedback style to enhance motivation.

[0063] As described above, the present invention realizes a system that provides users with inexpensive, highly accurate coaching that is available 24 hours a day, 365 days a year, allowing users to effectively improve their careers and performance.

[0064] The processing flow will be explained below.

[0065] Step 1:

[0066] The user uses the terminal to enter their basic information (name, occupation, goals, tasks, etc.). Specifically, the user enters the information into the input form on the terminal and sends the data by pressing the send button.

[0067] Step 2:

[0068] The user uploads not only text data but also media data such as images, audio, and video. Specifically, the user clicks the upload button, selects the required files from the file selection window, and completes the upload.

[0069] Step 3:

[0070] The terminal transmits the text data input by the user and the uploaded media data to the server. Specifically, the terminal creates an HTTP request and transmits the data to the server.

[0071] Step 4:

[0072] The server receives the data from the terminal and stores it in the database. Specifically, it analyzes the HTTP request, extracts the data, and executes the process to store it in the database.

[0073] Step 5:

[0074] The server analyzes the stored text data using natural language processing (NLP) technology, specifically using text analysis algorithms to extract and understand the key content of the goals and tasks.

[0075] Step 6:

[0076] The server analyzes the media data (images, audio, video) it receives. Specifically, it applies image recognition, audio analysis, and video analysis algorithms to classify and understand the content of the media data.

[0077] Step 7:

[0078] The server generates feedback based on the analysis results to help users achieve their goals and solve their problems. Specifically, it creates specific action plans and advice in the form of text or other media based on the analysis results.

[0079] Step 8:

[0080] The server sends the generated feedback and advice to the device. Specifically, the feedback data is sent as an HTTP response, and the device receives it.

[0081] Step 9:

[0082] The feedback received by the device is displayed to the user. Specifically, a UI (user interface) is constructed to display the feedback content on the screen in an appropriate format, allowing the user to easily access it.

[0083] Step 10:

[0084] The server periodically tracks the user's progress: each time the user uploads new data, it receives and analyzes it and evaluates the progress.

[0085] Step 11:

[0086] The user periodically uploads new data (e.g., a new presentation video). Specifically, the user repeatedly sends progress data from the device to the server.

[0087] Step 12:

[0088] The server compares the results of the analysis with the previous results and generates updated feedback, specifically, updates the feedback based on the new data and provides it to the user.

[0089] Step 13:

[0090] The server personalizes the feedback content and communication style based on the user's personality and preferences. Specifically, it selects the optimal feedback method based on the user's profile information and applies an individual approach.

[0091] This allows the system to provide users with efficient and personalized coaching, helping them improve their careers and performance.

[0092] Example 1

[0093] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0094] Conventional coaching systems require a lot of effort for users to input their goals and challenges, and provide general, blanket feedback that fails to adequately address individual needs. Furthermore, they lack a way to continuously track a user's progress or to personalize feedback based on personality and preferences. This makes it difficult to effectively support users in achieving their goals and resolving challenges.

[0095] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0096] In this invention, the server includes means for allowing a user to input goals and tasks and upload text data and media data, means for receiving and storing the input data, means for analyzing the received data and using natural language processing, image recognition, voice analysis, and video analysis to generate feedback for the user to achieve their goals and solve their tasks, and means for providing the generated feedback to the user, thereby enabling the provision of feedback that meets the individual needs of the user, continuous tracking of progress, and even personalized feedback.

[0097] A "user" is a person or organization that uses the system to input their goals and challenges and receive feedback.

[0098] A "goal" indicates the purpose or intention that a user is trying to achieve, and is information that is input into the system.

[0099] "Issues" indicate problems that users are trying to solve or areas that need improvement, and are information that is entered into the system.

[0100] "Text data" refers to text information entered by the user, including details of goals and tasks.

[0101] "Media data" refers to digital data such as images, audio, and video uploaded by users.

[0102] "Server" refers to the central system that receives, stores, analyzes, and generates and provides feedback on input data.

[0103] "Natural language processing" is a technique used by systems to analyze text data in order to understand the user's intent and meaning.

[0104] "Image recognition" is a technology that allows a system to analyze image data and understand its contents.

[0105] "Speech analysis" is a technology that allows a system to analyze voice data and evaluate its content and quality.

[0106] "Video analysis" is a technology that allows a system to analyze video data and evaluate its content and performance.

[0107] "Feedback" refers to information that provides evaluation and advice on the goals and tasks entered by the user.

[0108] "Personalization" refers to individually tailoring feedback content and communication style based on a user's personality and preferences.

[0109] "Tracking" means that the system continuously monitors and evaluates the user's progress.

[0110] The present invention is a system that provides effective feedback and coaching to businesspeople to help them achieve their goals and challenges. The system mainly includes the following components:

[0111] 1. User's device

[0112] Users use devices such as personal computers or smartphones to input goals and tasks as text data and upload media data such as images, audio, and video. This device acts as the user's interface, inputting data and receiving feedback.

[0113] Examples:

[0114] The user uses the smartphone app to enter their name (e.g., "Yamada Taro"), job title (e.g., "Sales Manager"), goal (e.g., "Success in a new project"), and task (e.g., "Improve presentation skills"), then presses the "Settings Complete" button. Next, they use the smartphone's camera function to record a video of a past presentation and upload it from the app.

[0115] 2. Server

[0116] The server receives text and media data entered from the user's device and stores it in a database. This data is analyzed using an analysis module within the server. Multimodal AI technologies such as natural language processing (NLP), image recognition, audio analysis, and video analysis are used for the analysis. Specific technologies used include Google Cloud Natural Language API and Google Cloud Video Intelligence API.

[0117] Examples:

[0118] The server analyzes the received text information and presentation video using the Google Cloud Natural Language API and Google Cloud Video Intelligence API. The server understands the user's intent and challenges from the text information, and evaluates pronunciation clarity, gestures, eye movements, etc. from the video data. The analysis results are stored in a dedicated table in the database.

[0119] 3. Providing Feedback

[0120] Based on the analysis results, the server generates a specific coaching plan for the user. The generated feedback and advice is sent to the user's device in text, audio, and video formats. The user can use this to improve their skills and solve problems.

[0121] Examples:

[0122] The server generates feedback data (e.g., "There were some parts where your pronunciation was unclear. Please try to speak more slowly.") and sends it to the user's smartphone. The user opens the app and checks the provided feedback.

[0123] The system based on the present invention is available to users 24 hours a day, 365 days a year, and provides inexpensive, highly accurate coaching, allowing users to effectively improve their careers and performance.

[0124] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0125] Step 1:

[0126] Entering user information and initial settings

[0127] Users start the application using their personal computer or smartphone, enter basic information (such as name, occupation, goals, and tasks), and upload image, audio, and video data as needed.

[0128] Input: Name, job title, goal, task, media data (images, audio, video)

[0129] Output: Packages the input data and converts it into a format that can be sent to the server.

[0130] Specific operation: The user opens the app on their smartphone and goes to the "Profile Settings" screen. They enter their name (e.g., "Yamada Taro"), job title (e.g., "Sales Manager"), goal (e.g., "Success in new projects"), and challenge (e.g., "Improve presentation skills") in the text boxes and press the "Settings Complete" button. Next, they record a video of a past presentation using the device's camera and upload it from within the app.

[0131] Step 2:

[0132] Sending and Receiving Data

[0133] The device compresses the input text and media data and sends it to the server via a secure protocol (e.g., HTTPS). The server receives this data and stores it in a dedicated database.

[0134] Input: Compressed text and media data

[0135] Output: User data stored in the server database

[0136] Specific operation: After the user presses the "Settings complete" button, the terminal compresses the data and sends it to the server using the SSL / TLS protocol. The server stores the received data in a database table dedicated to analysis.

[0137] Step 3:

[0138] Data analysis

[0139] The server uses a dedicated data analysis module to analyze the received data, using natural language processing (NLP), image recognition, audio analysis, and video analysis technologies.

[0140] Input: Stored text and media data

[0141] Output: Analysis results (data for creating feedback)

[0142] How it works: The server analyzes text information using the Google Cloud Natural Language API. It also analyzes uploaded presentation videos using the Google Cloud Video Intelligence API, evaluating pronunciation clarity, gestures, and eye movements. These results are stored in a database as analytical data that specifically indicates areas for improvement for the user.

[0143] Step 4:

[0144] Generate a coaching plan

[0145] Based on the analysis results, the server generates a specific coaching plan for the user to achieve their goals and solve their problems.

[0146] Input: Analysis result data

[0147] Output: Coaching plan (feedback, advice, specific action plan)

[0148] Specific operation: Based on the results of the analysis, the server uses the coaching plan generation module to create a specific action plan, such as "Practice speaking for 10 minutes every day to improve the fluency of your presentation." The generated plan is stored in a database table dedicated to user feedback.

[0149] Step 5:

[0150] Providing Feedback

[0151] The server sends the generated feedback and advice to the user's device, which the user can use to improve their skills and solve problems.

[0152] Input: Coaching plan data

[0153] Output: Feedback sent to the user device

[0154] Specific operation: The server generates feedback data (e.g., "There were some parts where your pronunciation was unclear. Please try to speak more slowly.") and sends it to the user's smartphone. The user opens the app and checks the provided feedback.

[0155] Step 6:

[0156] Tracking progress

[0157] The server continually tracks the user's progress and provides updated feedback as needed.

[0158] Input: Newly uploaded data (e.g., a new presentation video)

[0159] Output: Updated feedback

[0160] What it does: Users periodically upload new presentation videos from their devices. The server analyzes the videos, evaluates their progress, and provides new feedback.

[0161] Step 7:

[0162] Personalized and adaptive

[0163] The server personalizes the feedback and communication style based on the user's personality and preferences.

[0164] Input: Past feedback history and user response data

[0165] Output: Personalized feedback

[0166] Specific actions: The server analyzes the feedback history and the user's response patterns, generates personalized feedback such as "include positive encouragement" and "provide specific examples of success," and provides it to the user.

[0167] (Application example 1)

[0168] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0169] While factories need to improve work efficiency and accuracy, there is a lack of ways to coach individual workers in real time. It is also difficult to continuously track worker performance and provide individual feedback, making it difficult to maintain consistent work quality. Furthermore, while individualized instruction based on each worker's personality and preferences is required, current systems are nearly impossible to address.

[0170] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0171] In this invention, the server includes a means for users to input goals and tasks and upload text data and media data, a means for receiving and saving the input data, a means for analyzing the received data and generating feedback for users to achieve their goals and solve their tasks, and a means for recording factory workers' movements with a camera and providing feedback in real time. This allows for real-time coaching of factory workers' movements and the provision of individualized feedback. Furthermore, by continuously tracking the progress of workers and providing personalized guidance based on their individual personalities and preferences, it is possible to expect improved work quality and consistent performance.

[0172] A "user" is someone who uses this system to set goals and challenges and receives feedback and coaching based on those goals and challenges.

[0173] "Means for inputting goals and tasks" refers to a device or interface that allows a user to input their goals and current tasks to the system using text data or media data.

[0174] "Text data and media data" refers to digital data such as text information, images, audio, and video that users input or upload to the system.

[0175] "Means for receiving and storing" refers to a device or software that allows the system to receive text data and media data entered or uploaded by a user and store it in an appropriate location, such as a database.

[0176] "Means for analyzing and generating feedback" refers to devices or software that process received text data or media data using various analytical techniques and automatically generate feedback that helps the user achieve their goals or solve their problems.

[0177] The "means for providing feedback" refers to a device or software for transmitting the generated feedback to the user, and includes, for example, a smartphone or PC application.

[0178] "Means for recording the actions of factory workers" refers to devices or software that use cameras or sensors to photograph and record the actions of workers in a factory and send that data to the system.

[0179] "Means for providing real-time feedback" refers to a device or software that instantly analyzes recorded operational data of factory workers and provides immediate feedback.

[0180] A "means for continuous progress tracking" is a device or software that periodically tracks a user's or worker's achievements or work progress and generates new feedback based on that data.

[0181] "Personalization means" refers to a device or software that individually optimizes the content of feedback and communication style based on the user's personality and preferences.

[0182] This invention is a system for coaching factory workers' actions in real time to achieve efficient and accurate work. This system includes a series of processes in which users (workers) input their goals and tasks and receive feedback and coaching based on those goals and tasks.

[0183] Hardware and software used

[0184] 1. Hardware to be used

[0185] High-performance cameras: Record factory workers' movements in real time

[0186] Smart Glasses: Providing Visual Feedback to Workers

[0187] Factory robots: motion assistance and data collection

[0188] 2. Software to be used

[0189] Natural Language Processing (NLP) Library: SpaCy

[0190] Image recognition technology: TensorFlow + OpenCV

[0191] Server: Microsoft Azure

[0192] Performance analysis algorithm: Python + Scikit-learn

[0193] Program processing overview

[0194] 1. Data Collection:

[0195] The server collects real-time movement data of factory workers through high-performance cameras, including their work movements, speed, and accuracy.

[0196] 2. Data transmission:

[0197] The collected data is sent over a network to a server, which receives and securely stores the data.

[0198] 3. Data Analysis:

[0199] The server analyzes the received data using natural language processing with SpaCy and image recognition technology with TensorFlow and OpenCV. This analysis identifies areas that need improvement, such as work efficiency and accuracy.

[0200] 4. Coaching Plan Creation:

[0201] Based on the analysis results, the server uses Python and Scikit-learn to generate an individual coaching plan, which includes specific improvements and training methods to help workers achieve their goals efficiently.

[0202] 5. Providing Feedback:

[0203] The server provides the generated feedback to the worker in real time through smart glasses, which can also be provided as audio guidance or visual cues.

[0204] 6. Progress Tracking:

[0205] The server periodically tracks the worker's progress and updates the feedback as needed, allowing for sustained performance improvement.

[0206] Specific examples

[0207] For example, consider a factory worker performing an assembly task. A camera records the worker's movements in real time and sends the data to a server. The server analyzes the movement data and evaluates the worker's speed and frequency of mistakes. Based on the analysis results, the server generates a coaching plan that includes "points to improve work speed" and a "checklist for reducing mistakes." This feedback is provided to the worker in real time through the smart glasses, allowing the worker to immediately recognize areas for improvement and work more efficiently.

[0208] Prompt Sentence Examples

[0209] Analyze video and audio data of workers' movements and generate feedback to improve their performance. For example, evaluate their work speed, frequency of mistakes, and accuracy of work procedures, and create and provide a training plan based on that. Specifically, please advise on the following items:

[0210] 1. Measures to improve work speed

[0211] 2. Checkpoints to reduce mistakes

[0212] 3. Visual guides to help you follow the steps

[0213] This system is expected to significantly improve the efficiency and quality of factory workers' work, and by providing individual feedback to each worker, it will be possible to uniformly improve performance throughout the factory.

[0214] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0215] Program processing flow

[0216] Step 1:

[0217] Data collection

[0218] Actions: The terminal (factory robot) uses a high-performance camera to record the actions of factory workers in real time.

[0219] Input: Video and audio data of worker movements.

[0220] Output: Real-time recorded motion data.

[0221] How it works: Cameras track workers' movements and voices and record the data in storage.

[0222] Step 2:

[0223] Data transmission

[0224] Action: The device sends the recorded data to the server.

[0225] Input: Real-time recorded motion data.

[0226] Output: The operation data sent to the server.

[0227] Specific operation: Using the terminal's network module, the operation data is transferred to the specified directory on the server.

[0228] Step 3:

[0229] Data Receipt and Storage

[0230] Action: The server stores the received data.

[0231] Input: Operational data sent from the device.

[0232] Output: Saved operating data.

[0233] Specific operation: The server stores the received data in a database and prepares it for further analysis.

[0234] Step 4:

[0235] Data analysis

[0236] How it works: The server analyzes the stored data.

[0237] Input: Stored operating data.

[0238] Output: Analysis results (e.g., work speed, frequency of mistakes, accuracy assessment).

[0239] Specific behavior:

[0240] The server analyzes the motion data using TensorFlow + OpenCV.

[0241] Analyze the audio data using SpaCy.

[0242] Generate data-driven feedback using Scikit-learn models.

[0243] Step 5:

[0244] Coaching plan creation

[0245] How it works: The server generates a coaching plan based on the analysis results.

[0246] Input: Analysis results.

[0247] Output: Coaching plan.

[0248] Specific behavior:

[0249] Based on the analysis results, an individual coaching plan is created using a Python script.

[0250] The written coaching plan is saved in the database.

[0251] Step 6:

[0252] Providing Feedback

[0253] Action: The server provides the generated feedback to the user (worker).

[0254] Enter: Coaching Plan.

[0255] Output: Feedback provided to the user (worker).

[0256] Specific behavior:

[0257] The smart glasses provide real-time audio guidance and visual feedback to the user.

[0258] Feedback is projected directly into the user's field of vision, encouraging immediate improvement.

[0259] Step 7:

[0260] Progress Tracking

[0261] How it works: The server tracks the progress of the workers.

[0262] Input: New behavioral data collected periodically.

[0263] Output: Updated analysis results and feedback.

[0264] Specific behavior:

[0265] The server periodically receives and analyzes new data and updates the feedback based on the results.

[0266] Progress reports are generated and provided to users and administrators.

[0267] This process provides real-time feedback and continuous improvement to factory workers, and the personalized feedback is expected to improve overall efficiency and quality.

[0268] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0269] The present invention is a system that provides effective feedback and coaching to business people to help them achieve their goals and challenges, and further adjusts the feedback based on the user's emotions by combining it with an emotion engine. Specific embodiments for implementing the system are described below.

[0270] System configuration

[0271] The system mainly includes the following components:

[0272] 1. User's device

[0273] A device (such as a computer or smartphone) on which users can enter their goals and challenges and upload text and media data.

[0274] 2. Server

[0275] A central system that receives, stores, analyzes input data, and provides feedback to users.

[0276] 3. Emotion Engine

[0277] A system that recognizes emotions from the user's facial expressions, tone of voice, body movements, etc.

[0278] Program processing overview

[0279] 1. Entering user information and initial settings

[0280] A user uses a terminal to input basic information about themselves (such as name, occupation, goals, and tasks). This input includes not only text data, but also image, audio, and video data as needed. For example, a user who aims to succeed in a new project may set improving their presentation skills as a task and upload videos of past presentations to the system.

[0281] 2. Sending and Receiving Data

[0282] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[0283] 3. Data Analysis

[0284] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies, including natural language processing (NLP) of text data, as well as image recognition, speech analysis, and video analysis. For example, a presentation video can be analyzed to generate feedback on pronunciation and the flow of the presentation.

[0285] 4. Emotion analysis

[0286] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.). For example, it can recognize nervousness from the user's facial expressions during a presentation.

[0287] 5. Create a coaching plan

[0288] The server generates a specific coaching plan for the user based on the analysis results and the emotion engine results. This plan includes feedback, advice, and a specific action plan. For example, a training plan for improving presentation skills may be created, with specific points to improve and practice methods indicated. Advice based on the user's emotional state may also be included.

[0289] 6. Providing Feedback

[0290] The server generates feedback and advice, which is then sent to the device and provided to the user. The feedback is provided in the form of text, audio, video, etc., and the user can review and use it as appropriate. The user can also receive personalized feedback based on their emotional state.

[0291] 7. Track your progress

[0292] The server periodically tracks the user's progress: the user periodically uploads new data (e.g., a new presentation video), which the server analyzes and provides up-to-date feedback.

[0293] 8. Personalization and Adaptation

[0294] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state, resulting in more effective coaching. For example, if the user is nervous, the server can provide advice on how to relax.

[0295] This system allows users to receive highly efficient, personalized coaching at low cost, effectively improving their careers and performance.

[0296] The processing flow will be explained below.

[0297] Step 1:

[0298] The user uses the terminal to enter their basic information (name, occupation, goals, tasks, etc.). Specifically, the user enters the information into the input form on the terminal and sends the data by pressing the send button.

[0299] Step 2:

[0300] The user uploads not only text data but also media data such as images, audio, and video. Specifically, the user clicks the upload button, selects the required files from the file selection window, and completes the upload.

[0301] Step 3:

[0302] The terminal transmits the text data input by the user and the uploaded media data to the server. Specifically, the terminal creates an HTTP request and transmits the data to the server.

[0303] Step 4:

[0304] The server receives the data from the terminal and stores it in the database. Specifically, it analyzes the HTTP request, extracts the data, and executes the process to store it in the database.

[0305] Step 5:

[0306] The server analyzes the stored text data using natural language processing (NLP) technology, specifically using text analysis algorithms to extract and understand the key content of the goals and tasks.

[0307] Step 6:

[0308] The server analyzes the media data (images, audio, video) it receives. Specifically, it applies image recognition, audio analysis, and video analysis algorithms to classify and understand the content of the media data.

[0309] Step 7:

[0310] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.). For example, it can recognize nervousness from the user's facial expressions during a presentation.

[0311] Step 8:

[0312] The server integrates the results of the text data analysis, media data analysis, and emotion engine analysis to generate a specific coaching plan for the user to achieve their goals and solve their problems. Specifically, it creates feedback, advice, and a specific action plan in the form of text and media based on the analysis results.

[0313] Step 9:

[0314] The server sends the generated feedback and advice to the device. Specifically, the feedback data is sent as an HTTP response, and the device receives it.

[0315] Step 10:

[0316] The feedback received by the device is displayed to the user. Specifically, a UI (user interface) is constructed to display the feedback content on the screen in an appropriate format, allowing the user to easily access it.

[0317] Step 11:

[0318] The server periodically tracks the user's progress. Specifically, each time the user uploads new data (e.g., a new presentation video), it receives and analyzes it and evaluates the user's progress.

[0319] Step 12:

[0320] The user periodically uploads new data (e.g., a new presentation video). Specifically, the user repeatedly sends progress data from the device to the server.

[0321] Step 13:

[0322] The server compares the results of the analysis with the previous results and generates updated feedback, specifically, updates the feedback based on the new data and provides it to the user.

[0323] Step 14:

[0324] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state. Specifically, it selects the optimal feedback method based on the user's profile information and applies an individual approach.

[0325] This allows the system to provide users with efficient and personalized coaching to help them improve their careers and performance.

[0326] Example 2

[0327] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0328] Effective feedback and coaching are essential for modern businesspeople to achieve their goals and challenges. However, traditional systems struggle to provide personalized feedback in real time that reflects users' emotions and progress. Furthermore, the lack of detailed progress tracking and updated feedback makes it difficult for users to continuously improve.

[0329] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for a user to input goals and tasks and upload text data and media data, means for receiving and saving the input data, means for analyzing the received data and generating feedback for the user to achieve their goals and solve their tasks, means for recognizing the user's emotions from their facial expressions, voice, and body movements and adjusting the feedback based on the results, and means for providing the generated feedback to the user. This makes it possible to provide personalized feedback that takes the user's emotional state into consideration, allowing the user to efficiently continue making improvements toward achieving their goals.

[0330] A "user" is an individual or organization that uses the system to achieve a goal or task.

[0331] A "goal" is a specific outcome or objective that a user wishes to achieve.

[0332] A "challenge" is a problem or obstacle that a user must overcome.

[0333] "Text data" refers to information or content expressed in a written format.

[0334] "Media data" refers to visual or audible information such as images, audio, and video, and includes non-text data.

[0335] "Upload" refers to sending data from the user's terminal to the server.

[0336] A "server" is a central system that receives, stores, and analyzes input data, and generates and provides feedback to users.

[0337] The "emotion engine" is a system that recognizes a user's emotional state by analyzing their facial expressions, voice, and body movements.

[0338] "Feedback" refers to advice and evaluations to help users achieve their goals and solve their problems.

[0339] "Tracking" is the act of continuously monitoring and recording a user's progress and analyzing that data.

[0340] "Personalization" means tailoring feedback content and communication style based on the individual personality and preferences of each user.

[0341] MODE FOR CARRYING OUT THE INVENTION

[0342] System Program Overview

[0343] This invention is a system that provides effective feedback and coaching to businesspeople to help them achieve their goals and challenges. Furthermore, by combining it with an emotion engine, the system adjusts the feedback based on the user's emotions. The following specific hardware and software are used to implement this system:

[0344] Hardware and software used

[0345] 1. On the user's device:

[0346] PC, smartphone, etc. Users can input their goals and tasks and upload text and media data.

[0347] 2. Server:

[0348] It acts as a central system, receiving, storing, and analyzing input data and providing feedback to users.

[0349] 3. Emotion Engine:

[0350] A system that recognizes emotions from a user's facial expressions, tone of voice, body movements, etc. For example, it uses facial expression recognition technology and voice analysis technology.

[0351] Program processing overview

[0352] Entering user information and initial settings

[0353] A user uses a terminal to input basic information about themselves (such as name, occupation, goals, and tasks). This input includes not only text data, but also image, audio, and video data as needed. For example, a user who aims to succeed in a new project may set improving their presentation skills as a task and upload videos of past presentations to the system.

[0354] Sending and Receiving Data

[0355] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[0356] Data analysis

[0357] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies, including natural language processing (NLP) of text data, as well as image recognition, speech analysis, and video analysis. For example, a presentation video can be analyzed to generate feedback on pronunciation and the flow of the presentation.

[0358] Emotion analysis

[0359] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.). For example, it can recognize nervousness from the user's facial expressions during a presentation.

[0360] Generate a coaching plan

[0361] The server generates a specific coaching plan for the user based on the analysis results and the emotion engine results. This plan includes feedback, advice, and a specific action plan. For example, a training plan for improving presentation skills could be created, with specific points to improve and practice methods.

[0362] Providing Feedback

[0363] The server generates feedback and advice, which is then sent to the device and provided to the user. Feedback is provided in the form of text, audio, video, etc., and the user can review and use it as appropriate. The user can also receive personalized feedback based on their emotional state.

[0364] Tracking progress

[0365] The server periodically tracks the user's progress: the user periodically uploads new data (e.g. a new presentation video), which the server analyzes and provides up-to-date feedback.

[0366] Personalized and adaptive

[0367] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state, resulting in more effective coaching. For example, if the user is nervous, the server can provide advice on how to relax.

[0368] Specific examples and examples of input prompts to the generative AI model

[0369] Specific examples

[0370] 1. A user aims to improve their presentation skills in order to succeed in a new project.

[0371] 2. The user uploads past presentation videos to the system from their device.

[0372] 3. The server receives the video and begins analyzing it, generating feedback using NLP, image recognition, and voice analysis technologies.

[0373] 4. The emotion engine recognizes the user's state of tension from the video and sends the emotional state to the server.

[0374] 5. The server generates feedback and gives specific advice, such as "Practice speaking slowly to improve your pronunciation. Take deep breaths when you feel nervous."

[0375] 6. When the user uploads a new video, feedback will be provided again.

[0376] Example of input prompt for generative AI model

[0377] Analyze the following data:

[0378] Name: Yamada Taro

[0379] Job type: Sales

[0380] Goal: New project success

[0381] Challenge: Improve presentation skills

[0382] Uploaded data: Past presentation videos

[0383] Use this data to generate feedback and a specific coaching plan, including advice on what to do if you're nervous.

[0384] This system allows users to receive highly efficient, personalized coaching at low cost, effectively improving their careers and performance.

[0385] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0386] Step 1: Entering user information and initial settings

[0387] A user logs into the system using a terminal and enters their basic information (name, job title, goals, challenges, etc.). This input includes not only text data, but also image, audio, video, and other data as needed. For example, a user whose goal is to succeed in a new project may set improving their presentation skills as a challenge and upload videos of past presentations to the system.

[0388] Input: User basic information, text data, media data

[0389] Output: Save input data

[0390] Specific behavior:

[0391] A user accesses the system's login page and logs in by entering authentication information (user name and password).

[0392] After logging in, a basic information input form is displayed and the user enters the required data.

[0393] Enter your goals and objectives, and upload media data (such as presentation videos).

[0394] Step 2: Sending and Receiving Data

[0395] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[0396] Input: Data entered by the user

[0397] Output: Save data to database

[0398] Specific behavior:

[0399] When the user clicks the "Send" button, the terminal sends the data to the server.

[0400] The server processes the received data in real time and stores it in a database.

[0401] Step 3: Data analysis

[0402] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies such as image recognition, voice analysis, and video analysis, as well as natural language processing (NLP) of text data.

[0403] Input: User text data, media data

[0404] Output: Feedback generation

[0405] Specific behavior:

[0406] The server launches an NLP engine to analyze the meaning of the text data.

[0407] The video analysis engine analyzes the uploaded presentation video frame by frame and evaluates the user's speaking speed and clarity.

[0408] The voice analysis engine analyzes the tone and pace of the user's voice.

[0409] Step 4: Sentiment Analysis

[0410] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.).

[0411] Input: User's video data, audio data

[0412] Output: Emotion recognition result

[0413] Specific behavior:

[0414] The emotion engine uses facial expression recognition technology to analyze the user's facial features in the video.

[0415] The server receives the emotion analysis results and uses them to generate feedback.

[0416] Step 5: Create a coaching plan

[0417] The server generates a specific coaching plan for the user based on the results of the data analysis and the sentiment analysis, which includes feedback, advice, and a specific action plan.

[0418] Input: Data analysis results, emotion recognition results

[0419] Output: Coaching plan generated

[0420] Specific behavior:

[0421] The server generates detailed feedback on how to improve and practice your presentation.

[0422] The server will also provide relaxation techniques and advice to help relieve tension.

[0423] Step 6: Provide feedback

[0424] The server generates feedback and advice, which is sent to the device and provided to the user in the form of text, audio, video, etc.

[0425] Input: Generated feedback

[0426] Output: Providing feedback

[0427] Specific behavior:

[0428] The server generates feedback and sends it to the user's device as a push notification.

[0429] The feedback content is displayed on the terminal screen and the user can check it.

[0430] Step 7: Track your progress

[0431] The server periodically tracks the user's progress: the user periodically uploads new data (e.g. a new presentation video), which the server re-analyzes and provides updated feedback.

[0432] Input: User's new data

[0433] Output: Updated feedback

[0434] Specific behavior:

[0435] The server compares the user's past feedback with their current progress and generates an updated report.

[0436] When a user uploads a new presentation video, the server compares it with previous analysis results and identifies areas for improvement.

[0437] Step 8: Personalize and adapt

[0438] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state, resulting in more effective coaching.

[0439] Input: User's personality profile, emotion recognition results

[0440] Output: Personalized feedback

[0441] Specific behavior:

[0442] The server adjusts the feedback content based on the user's personality profile and past feedback results.

[0443] Providing feedback based on emotional state (e.g., suggesting ways to relax if you're feeling tense).

[0444] (Application example 2)

[0445] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0446] There is a need for systems that provide feedback and coaching to improve the work efficiency of robotic workers in factories, reduce errors, and maintain optimal work conditions. However, conventional systems lack sufficient analysis of motion and sensor data, and are unable to provide appropriate feedback based on emotional states. This results in poor robot performance and has a negative impact on work efficiency and quality.

[0447] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0448] In this invention, the server includes means for a user to input goals and tasks and upload text data and media data, means for receiving and saving the input data, means for analyzing the received data and generating feedback for the user to achieve their goals and solve their tasks, means for providing the generated feedback to the user, means for analyzing video of the robot's work and sensor data and generating feedback for improving work efficiency and reducing errors, and means for performing emotional analysis based on the robot's sensor data and providing appropriate feedback according to the robot's emotional state. This makes it possible to analyze the robot's operating status in real time and provide optimal feedback and coaching.

[0449] A "user" is someone who uses the system or provides data to help the robotic worker achieve its goals and solve its problems.

[0450] "Goal" refers to the result or purpose that a user aims to achieve.

[0451] A "challenge" refers to a problem or obstacle that a user must overcome in the process of achieving a goal.

[0452] "Text data" refers to data that includes character information entered by the user.

[0453] "Media data" refers to data in various formats such as images, audio, and video.

[0454] "Reception" refers to the process in which the server acquires data sent from the user terminal.

[0455] "Storage" refers to recording the received data in a database or storage within the server.

[0456] "Analysis" is the process of extracting and evaluating information based on the received data to help the user achieve their goals and solve their problems.

[0457] "Feedback" refers to information or advice returned to the user based on the analysis results.

[0458] "Robot work video" is video data that records the movements of robots operating in a factory.

[0459] "Sensor data" refers to data such as temperature, vibration, and position obtained from sensors installed on the robot.

[0460] "Emotion analysis" is the process of inferring a robot's emotional state based on its movements and sensor data.

[0461] "Appropriate feedback" refers to the most effective advice and information based on the robot's state, taking into account the results of emotion analysis.

[0462] This invention is a system for improving the work efficiency and reducing errors of robot workers in factories. The system includes means for a user to input goals and tasks and upload text data and media data, means for receiving and saving the data, means for analyzing the received data, means for providing generated feedback to the user, means for analyzing video of the robot's work and sensor data to generate feedback for improving work efficiency and reducing errors, and means for performing emotional analysis and providing feedback according to the emotional state.

[0463] The server receives data input from the user's device (for example, information about the goals and challenges of robot workers in a factory, along with the associated text and media data) and stores it in a database. Software libraries such as TensorFlow and OpenCV are used to analyze the received data. These libraries are used to analyze motion in videos and sensor data, and generate the feedback necessary to help the user achieve their goals and solve their problems.

[0464] Furthermore, the server uses an emotion analysis engine to estimate the robot's emotional state (e.g., stress or fatigue) using video and sensor data from the robot's work. The emotion analysis engine evaluates the robot's movements and sensor data based on a specific emotion model and adjusts the feedback content. The generated feedback may include specific improvements, training modules, or maintenance schedules to improve work efficiency.

[0465] As a concrete example of this system, imagine a robot working on a conveyor line. Videos of the robot's work are recorded and sensor data (such as vibration and temperature data) is acquired. The server analyzes this data and provides feedback suggesting techniques to optimize the robot's movements, or, if high stress is detected from the sensor data, feedback suggesting when maintenance should be performed.

[0466] An example prompt might look like this:

[0467] "What operational improvements are needed to ensure that this conveyor line robot operates efficiently? Also, what maintenance is appropriate if it is experiencing fatigue or stress?"

[0468] This allows robotic workers in factories to receive optimal feedback and coaching in real time, improving work efficiency and reducing errors.

[0469] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0470] Step 1:

[0471] The server provides a means for users to input goals and tasks and upload text and media data. This allows users to set specific goals and tasks for the system and send data related to them. Input data could be, for example, video of robots working in a factory or sensor data. This input data is sent to the server and stored.

[0472] Step 2:

[0473] The server has a means for receiving and storing the input data. The received data includes text data (goals and tasks), video data (videos of the robot working), and sensor data (temperature, vibration, etc.). The received data is recorded in a database for subsequent analysis.

[0474] Step 3:

[0475] The server analyzes the received data. Specifically, it uses TensorFlow and OpenCV to analyze the movements in the video and extract important movement information from each frame. It also analyzes sensor data to determine the robot's operating status and detect abnormalities. This allows it to identify, for example, areas where the robot's movements are lagging or where abnormal vibrations are occurring.

[0476] Step 4:

[0477] The server provides the generated feedback to the user. Based on the analysis results, it generates specific advice on operation optimization techniques and areas for improvement. For example, if a specific operation is slow, it will suggest techniques to optimize that operation, and if an abnormality is detected, it will suggest maintenance.

[0478] Step 5:

[0479] The server performs emotion analysis based on the robot's work video and sensor data. The emotion analysis engine evaluates the video and sensor data to estimate the robot's emotional state (such as stress or fatigue). This allows it to provide individual feedback based on its condition.

[0480] Step 6:

[0481] The server then provides final feedback to the user. Taking into account the results of sentiment analysis, the server provides more personalized feedback to the user. For example, if high stress is detected, the server provides feedback recommending regular rest and maintenance.

[0482] Step 7:

[0483] Users receive feedback provided by the server and use it to improve the robot's work and for maintenance. This improves the robot's work efficiency and reduces errors. For example, by implementing the proposed motion optimization technology, the speed and accuracy of work can be improved. In addition, by performing regular maintenance, the robot's uptime can be maximized.

[0484] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0485] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0486] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0487] [Second embodiment]

[0488] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0489] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0490] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0491] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0492] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0493] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0494] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0495] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0496] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0497] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0498] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0499] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0500] The present invention is a system that provides effective feedback and coaching to business people to help them achieve their goals and challenges. Specific embodiments for implementing the system will be described below.

[0501] System configuration

[0502] The system mainly includes the following components:

[0503] 1. User's device

[0504] A device (such as a computer or smartphone) on which users can enter their goals and challenges and upload text and media data.

[0505] 2. Server

[0506] A central system that receives, stores, analyzes input data, and provides feedback to users.

[0507] Program processing overview

[0508] 1. Entering user information and initial settings

[0509] A user uses a terminal to input their basic information (such as name, occupation, goals, and tasks). This input includes not only text data, but also image, audio, and video data as needed. For example, a user aiming to succeed in a new project may set improving their presentation skills as a task and upload videos of past presentations to the system.

[0510] 2. Sending and Receiving Data

[0511] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[0512] 3. Data Analysis

[0513] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies, including natural language processing (NLP) of text data, as well as image recognition, speech analysis, and video analysis. For example, a presentation video can be analyzed to generate feedback on pronunciation and the flow of the presentation.

[0514] 4. Create a coaching plan

[0515] Based on the analysis results, the server generates a specific coaching plan for the user, which includes feedback, advice, and a concrete action plan. For example, a training plan for improving presentation skills could be created, with specific points to improve and practice methods.

[0516] 5. Providing Feedback

[0517] The server then sends the generated feedback and advice to the terminal and provides it to the user. The feedback is provided in the form of text, audio, video, etc., and the user can check and use it as appropriate.

[0518] 6. Track your progress

[0519] The server periodically tracks the user's progress: the user periodically uploads new data (e.g., a new presentation video), which the server analyzes and provides up-to-date feedback.

[0520] 7. Personalization and Adaptation

[0521] The server personalizes the feedback and communication style based on the user's personality and preferences, resulting in more effective coaching. For example, it adopts a positive feedback style to enhance motivation.

[0522] As described above, the present invention realizes a system that provides users with inexpensive, highly accurate coaching that is available 24 hours a day, 365 days a year, allowing users to effectively improve their careers and performance.

[0523] The processing flow will be explained below.

[0524] Step 1:

[0525] The user uses the terminal to enter their basic information (name, occupation, goals, tasks, etc.). Specifically, the user enters the information into the input form on the terminal and sends the data by pressing the send button.

[0526] Step 2:

[0527] The user uploads not only text data but also media data such as images, audio, and video. Specifically, the user clicks the upload button, selects the required files from the file selection window, and completes the upload.

[0528] Step 3:

[0529] The terminal transmits the text data input by the user and the uploaded media data to the server. Specifically, the terminal creates an HTTP request and transmits the data to the server.

[0530] Step 4:

[0531] The server receives the data from the terminal and stores it in the database. Specifically, it analyzes the HTTP request, extracts the data, and executes the process to store it in the database.

[0532] Step 5:

[0533] The server analyzes the stored text data using natural language processing (NLP) technology, specifically using text analysis algorithms to extract and understand the key content of the goals and tasks.

[0534] Step 6:

[0535] The server analyzes the media data (images, audio, video) it receives. Specifically, it applies image recognition, audio analysis, and video analysis algorithms to classify and understand the content of the media data.

[0536] Step 7:

[0537] The server generates feedback based on the analysis results to help users achieve their goals and solve their problems. Specifically, it creates specific action plans and advice in the form of text or other media based on the analysis results.

[0538] Step 8:

[0539] The server sends the generated feedback and advice to the device. Specifically, the feedback data is sent as an HTTP response, and the device receives it.

[0540] Step 9:

[0541] The feedback received by the device is displayed to the user. Specifically, a UI (user interface) is constructed to display the feedback content on the screen in an appropriate format, allowing the user to easily access it.

[0542] Step 10:

[0543] The server periodically tracks the user's progress: each time the user uploads new data, it receives and analyzes it and evaluates the progress.

[0544] Step 11:

[0545] The user periodically uploads new data (e.g., a new presentation video). Specifically, the user repeatedly sends progress data from the device to the server.

[0546] Step 12:

[0547] The server compares the results of the analysis with the previous results and generates updated feedback, specifically, updates the feedback based on the new data and provides it to the user.

[0548] Step 13:

[0549] The server personalizes the feedback content and communication style based on the user's personality and preferences. Specifically, it selects the optimal feedback method based on the user's profile information and applies an individual approach.

[0550] This allows the system to provide users with efficient and personalized coaching, helping them improve their careers and performance.

[0551] Example 1

[0552] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0553] Conventional coaching systems require a lot of effort for users to input their goals and challenges, and provide general, blanket feedback that fails to adequately address individual needs. Furthermore, they lack a way to continuously track a user's progress or to personalize feedback based on personality and preferences. This makes it difficult to effectively support users in achieving their goals and resolving challenges.

[0554] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0555] In this invention, the server includes means for allowing a user to input goals and tasks and upload text data and media data, means for receiving and storing the input data, means for analyzing the received data and using natural language processing, image recognition, voice analysis, and video analysis to generate feedback for the user to achieve their goals and solve their tasks, and means for providing the generated feedback to the user, thereby enabling the provision of feedback that meets the individual needs of the user, continuous tracking of progress, and even personalized feedback.

[0556] A "user" is a person or organization that uses the system to input their goals and challenges and receive feedback.

[0557] A "goal" indicates the purpose or intention that a user is trying to achieve, and is information that is input into the system.

[0558] "Issues" indicate problems that users are trying to solve or areas that need improvement, and are information that is entered into the system.

[0559] "Text data" refers to text information entered by the user, including details of goals and tasks.

[0560] "Media data" refers to digital data such as images, audio, and video uploaded by users.

[0561] "Server" refers to the central system that receives, stores, analyzes, and generates and provides feedback on input data.

[0562] "Natural language processing" is a technique used by systems to analyze text data in order to understand the user's intent and meaning.

[0563] "Image recognition" is a technology that allows a system to analyze image data and understand its contents.

[0564] "Speech analysis" is a technology that allows a system to analyze voice data and evaluate its content and quality.

[0565] "Video analysis" is a technology that allows a system to analyze video data and evaluate its content and performance.

[0566] "Feedback" refers to information that provides evaluation and advice on the goals and tasks entered by the user.

[0567] "Personalization" refers to individually tailoring feedback content and communication style based on a user's personality and preferences.

[0568] "Tracking" means that the system continuously monitors and evaluates the user's progress.

[0569] The present invention is a system that provides effective feedback and coaching to businesspeople to help them achieve their goals and challenges. The system mainly includes the following components:

[0570] 1. User's device

[0571] Users use devices such as personal computers or smartphones to input goals and tasks as text data and upload media data such as images, audio, and video. This device acts as the user's interface, inputting data and receiving feedback.

[0572] Examples:

[0573] The user uses the smartphone app to enter their name (e.g., "Yamada Taro"), job title (e.g., "Sales Manager"), goal (e.g., "Success in a new project"), and task (e.g., "Improve presentation skills"), then presses the "Settings Complete" button. Next, they use the smartphone's camera function to record a video of a past presentation and upload it from the app.

[0574] 2. Server

[0575] The server receives text and media data entered from the user's device and stores it in a database. This data is analyzed using an analysis module within the server. Multimodal AI technologies such as natural language processing (NLP), image recognition, audio analysis, and video analysis are used for the analysis. Specific technologies used include Google Cloud Natural Language API and Google Cloud Video Intelligence API.

[0576] Examples:

[0577] The server analyzes the received text information and presentation video using the Google Cloud Natural Language API and Google Cloud Video Intelligence API. The server understands the user's intent and challenges from the text information, and evaluates pronunciation clarity, gestures, eye movements, etc. from the video data. The analysis results are stored in a dedicated table in the database.

[0578] 3. Providing Feedback

[0579] Based on the analysis results, the server generates a specific coaching plan for the user. The generated feedback and advice is sent to the user's device in text, audio, and video formats. The user can use this to improve their skills and solve problems.

[0580] Examples:

[0581] The server generates feedback data (e.g., "There were some parts where your pronunciation was unclear. Please try to speak more slowly.") and sends it to the user's smartphone. The user opens the app and checks the provided feedback.

[0582] The system based on the present invention is available to users 24 hours a day, 365 days a year, and provides inexpensive, highly accurate coaching, allowing users to effectively improve their careers and performance.

[0583] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0584] Step 1:

[0585] Entering user information and initial settings

[0586] Users start the application using their personal computer or smartphone, enter basic information (such as name, occupation, goals, and tasks), and upload image, audio, and video data as needed.

[0587] Input: Name, job title, goal, task, media data (images, audio, video)

[0588] Output: Packages the input data and converts it into a format that can be sent to the server.

[0589] Specific operation: The user opens the app on their smartphone and goes to the "Profile Settings" screen. They enter their name (e.g., "Yamada Taro"), job title (e.g., "Sales Manager"), goal (e.g., "Success in new projects"), and challenge (e.g., "Improve presentation skills") in the text boxes and press the "Settings Complete" button. Next, they record a video of a past presentation using the device's camera and upload it from within the app.

[0590] Step 2:

[0591] Sending and Receiving Data

[0592] The device compresses the input text and media data and sends it to the server via a secure protocol (e.g., HTTPS). The server receives this data and stores it in a dedicated database.

[0593] Input: Compressed text and media data

[0594] Output: User data stored in the server database

[0595] Specific operation: After the user presses the "Settings complete" button, the terminal compresses the data and sends it to the server using the SSL / TLS protocol. The server stores the received data in a database table dedicated to analysis.

[0596] Step 3:

[0597] Data analysis

[0598] The server uses a dedicated data analysis module to analyze the received data, using natural language processing (NLP), image recognition, audio analysis, and video analysis technologies.

[0599] Input: Stored text and media data

[0600] Output: Analysis results (data for creating feedback)

[0601] How it works: The server analyzes text information using the Google Cloud Natural Language API. It also analyzes uploaded presentation videos using the Google Cloud Video Intelligence API, evaluating pronunciation clarity, gestures, and eye movements. These results are stored in a database as analytical data that specifically indicates areas for improvement for the user.

[0602] Step 4:

[0603] Generate a coaching plan

[0604] Based on the analysis results, the server generates a specific coaching plan for the user to achieve their goals and solve their problems.

[0605] Input: Analysis result data

[0606] Output: Coaching plan (feedback, advice, specific action plan)

[0607] Specific operation: Based on the results of the analysis, the server uses the coaching plan generation module to create a specific action plan, such as "Practice speaking for 10 minutes every day to improve the fluency of your presentation." The generated plan is stored in a database table dedicated to user feedback.

[0608] Step 5:

[0609] Providing Feedback

[0610] The server sends the generated feedback and advice to the user's device, which the user can use to improve their skills and solve problems.

[0611] Input: Coaching plan data

[0612] Output: Feedback sent to the user device

[0613] Specific operation: The server generates feedback data (e.g., "There were some parts where your pronunciation was unclear. Please try to speak more slowly.") and sends it to the user's smartphone. The user opens the app and checks the provided feedback.

[0614] Step 6:

[0615] Tracking progress

[0616] The server continually tracks the user's progress and provides updated feedback as needed.

[0617] Input: Newly uploaded data (e.g., a new presentation video)

[0618] Output: Updated feedback

[0619] What it does: Users periodically upload new presentation videos from their devices. The server analyzes the videos, evaluates their progress, and provides new feedback.

[0620] Step 7:

[0621] Personalized and adaptive

[0622] The server personalizes the feedback and communication style based on the user's personality and preferences.

[0623] Input: Past feedback history and user response data

[0624] Output: Personalized feedback

[0625] Specific actions: The server analyzes the feedback history and the user's response patterns, generates personalized feedback such as "include positive encouragement" and "provide specific examples of success," and provides it to the user.

[0626] (Application example 1)

[0627] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0628] While factories need to improve work efficiency and accuracy, there is a lack of ways to coach individual workers in real time. It is also difficult to continuously track worker performance and provide individual feedback, making it difficult to maintain consistent work quality. Furthermore, while individualized instruction based on each worker's personality and preferences is required, current systems are nearly impossible to address.

[0629] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0630] In this invention, the server includes a means for users to input goals and tasks and upload text data and media data, a means for receiving and saving the input data, a means for analyzing the received data and generating feedback for users to achieve their goals and solve their tasks, and a means for recording factory workers' movements with a camera and providing feedback in real time. This allows for real-time coaching of factory workers' movements and the provision of individualized feedback. Furthermore, by continuously tracking the progress of workers and providing personalized guidance based on their individual personalities and preferences, it is possible to expect improved work quality and consistent performance.

[0631] A "user" is someone who uses this system to set goals and challenges and receives feedback and coaching based on those goals and challenges.

[0632] "Means for inputting goals and tasks" refers to a device or interface that allows a user to input their goals and current tasks to the system using text data or media data.

[0633] "Text data and media data" refers to digital data such as text information, images, audio, and video that users input or upload to the system.

[0634] "Means for receiving and storing" refers to a device or software that allows the system to receive text data and media data entered or uploaded by a user and store it in an appropriate location, such as a database.

[0635] "Means for analyzing and generating feedback" refers to devices or software that process received text data or media data using various analytical techniques and automatically generate feedback that helps the user achieve their goals or solve their problems.

[0636] The "means for providing feedback" refers to a device or software for transmitting the generated feedback to the user, and includes, for example, a smartphone or PC application.

[0637] "Means for recording the actions of factory workers" refers to devices or software that use cameras or sensors to photograph and record the actions of workers in a factory and send that data to the system.

[0638] "Means for providing real-time feedback" refers to a device or software that instantly analyzes recorded operational data of factory workers and provides immediate feedback.

[0639] A "means for continuous progress tracking" is a device or software that periodically tracks a user's or worker's achievements or work progress and generates new feedback based on that data.

[0640] "Personalization means" refers to a device or software that individually optimizes the content of feedback and communication style based on the user's personality and preferences.

[0641] This invention is a system for coaching factory workers' actions in real time to achieve efficient and accurate work. This system includes a series of processes in which users (workers) input their goals and tasks and receive feedback and coaching based on those goals and tasks.

[0642] Hardware and software used

[0643] 1. Hardware to be used

[0644] High-performance cameras: Record factory workers' movements in real time

[0645] Smart Glasses: Providing Visual Feedback to Workers

[0646] Factory robots: motion assistance and data collection

[0647] 2. Software to be used

[0648] Natural Language Processing (NLP) Library: SpaCy

[0649] Image recognition technology: TensorFlow + OpenCV

[0650] Server: Microsoft Azure

[0651] Performance analysis algorithm: Python + Scikit-learn

[0652] Program processing overview

[0653] 1. Data Collection:

[0654] The server collects real-time movement data of factory workers through high-performance cameras, including their work movements, speed, and accuracy.

[0655] 2. Data transmission:

[0656] The collected data is sent over a network to a server, which receives and securely stores the data.

[0657] 3. Data Analysis:

[0658] The server analyzes the received data using natural language processing with SpaCy and image recognition technology with TensorFlow and OpenCV. This analysis identifies areas that need improvement, such as work efficiency and accuracy.

[0659] 4. Coaching Plan Creation:

[0660] Based on the analysis results, the server uses Python and Scikit-learn to generate an individual coaching plan, which includes specific improvements and training methods to help workers achieve their goals efficiently.

[0661] 5. Providing Feedback:

[0662] The server provides the generated feedback to the worker in real time through smart glasses, which can also be provided as audio guidance or visual cues.

[0663] 6. Progress Tracking:

[0664] The server periodically tracks the worker's progress and updates the feedback as needed, allowing for sustained performance improvement.

[0665] Specific examples

[0666] For example, consider a factory worker performing an assembly task. A camera records the worker's movements in real time and sends the data to a server. The server analyzes the movement data and evaluates the worker's speed and frequency of mistakes. Based on the analysis results, the server generates a coaching plan that includes "points to improve work speed" and a "checklist for reducing mistakes." This feedback is provided to the worker in real time through the smart glasses, allowing the worker to immediately recognize areas for improvement and work more efficiently.

[0667] Prompt Sentence Examples

[0668] Analyze video and audio data of workers' movements and generate feedback to improve their performance. For example, evaluate their work speed, frequency of mistakes, and accuracy of work procedures, and create and provide a training plan based on that. Specifically, please advise on the following items:

[0669] 1. Measures to improve work speed

[0670] 2. Checkpoints to reduce mistakes

[0671] 3. Visual guides to help you follow the steps

[0672] This system is expected to significantly improve the efficiency and quality of factory workers' work, and by providing individual feedback to each worker, it will be possible to uniformly improve performance throughout the factory.

[0673] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0674] Program processing flow

[0675] Step 1:

[0676] Data collection

[0677] Actions: The terminal (factory robot) uses a high-performance camera to record the actions of factory workers in real time.

[0678] Input: Video and audio data of worker movements.

[0679] Output: Real-time recorded motion data.

[0680] How it works: Cameras track workers' movements and voices and record the data in storage.

[0681] Step 2:

[0682] Data transmission

[0683] Action: The device sends the recorded data to the server.

[0684] Input: Real-time recorded motion data.

[0685] Output: The operation data sent to the server.

[0686] Specific operation: Using the terminal's network module, the operation data is transferred to the specified directory on the server.

[0687] Step 3:

[0688] Data Receipt and Storage

[0689] Action: The server stores the received data.

[0690] Input: Operational data sent from the device.

[0691] Output: Saved operating data.

[0692] Specific operation: The server stores the received data in a database and prepares it for further analysis.

[0693] Step 4:

[0694] Data analysis

[0695] How it works: The server analyzes the stored data.

[0696] Input: Stored operating data.

[0697] Output: Analysis results (e.g., work speed, frequency of mistakes, accuracy assessment).

[0698] Specific behavior:

[0699] The server analyzes the motion data using TensorFlow + OpenCV.

[0700] Analyze the audio data using SpaCy.

[0701] Generate data-driven feedback using Scikit-learn models.

[0702] Step 5:

[0703] Coaching plan creation

[0704] How it works: The server generates a coaching plan based on the analysis results.

[0705] Input: Analysis results.

[0706] Output: Coaching plan.

[0707] Specific behavior:

[0708] Based on the analysis results, an individual coaching plan is created using a Python script.

[0709] The written coaching plan is saved in the database.

[0710] Step 6:

[0711] Providing Feedback

[0712] Action: The server provides the generated feedback to the user (worker).

[0713] Enter: Coaching Plan.

[0714] Output: Feedback provided to the user (worker).

[0715] Specific behavior:

[0716] The smart glasses provide real-time audio guidance and visual feedback to the user.

[0717] Feedback is projected directly into the user's field of vision, encouraging immediate improvement.

[0718] Step 7:

[0719] Progress Tracking

[0720] How it works: The server tracks the progress of the workers.

[0721] Input: New behavioral data collected periodically.

[0722] Output: Updated analysis results and feedback.

[0723] Specific behavior:

[0724] The server periodically receives and analyzes new data and updates the feedback based on the results.

[0725] Progress reports are generated and provided to users and administrators.

[0726] This process provides real-time feedback and continuous improvement to factory workers, and the personalized feedback is expected to improve overall efficiency and quality.

[0727] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0728] The present invention is a system that provides effective feedback and coaching to business people to help them achieve their goals and challenges, and further adjusts the feedback based on the user's emotions by combining it with an emotion engine. Specific embodiments for implementing the system are described below.

[0729] System configuration

[0730] The system mainly includes the following components:

[0731] 1. User's device

[0732] A device (such as a computer or smartphone) on which users can enter their goals and challenges and upload text and media data.

[0733] 2. Server

[0734] A central system that receives, stores, analyzes input data, and provides feedback to users.

[0735] 3. Emotion Engine

[0736] A system that recognizes emotions from the user's facial expressions, tone of voice, body movements, etc.

[0737] Program processing overview

[0738] 1. Entering user information and initial settings

[0739] A user uses a terminal to input basic information about themselves (such as name, occupation, goals, and tasks). This input includes not only text data, but also image, audio, and video data as needed. For example, a user who aims to succeed in a new project may set improving their presentation skills as a task and upload videos of past presentations to the system.

[0740] 2. Sending and Receiving Data

[0741] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[0742] 3. Data Analysis

[0743] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies, including natural language processing (NLP) of text data, as well as image recognition, speech analysis, and video analysis. For example, a presentation video can be analyzed to generate feedback on pronunciation and the flow of the presentation.

[0744] 4. Emotion analysis

[0745] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.). For example, it can recognize nervousness from the user's facial expressions during a presentation.

[0746] 5. Create a coaching plan

[0747] The server generates a specific coaching plan for the user based on the analysis results and the emotion engine results. This plan includes feedback, advice, and a specific action plan. For example, a training plan for improving presentation skills may be created, with specific points to improve and practice methods indicated. Advice based on the user's emotional state may also be included.

[0748] 6. Providing Feedback

[0749] The server generates feedback and advice, which is then sent to the device and provided to the user. The feedback is provided in the form of text, audio, video, etc., and the user can review and use it as appropriate. The user can also receive personalized feedback based on their emotional state.

[0750] 7. Track your progress

[0751] The server periodically tracks the user's progress: the user periodically uploads new data (e.g., a new presentation video), which the server analyzes and provides up-to-date feedback.

[0752] 8. Personalization and Adaptation

[0753] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state, resulting in more effective coaching. For example, if the user is nervous, the server can provide advice on how to relax.

[0754] This system allows users to receive highly efficient, personalized coaching at low cost, effectively improving their careers and performance.

[0755] The processing flow will be explained below.

[0756] Step 1:

[0757] The user uses the terminal to enter their basic information (name, occupation, goals, tasks, etc.). Specifically, the user enters the information into the input form on the terminal and sends the data by pressing the send button.

[0758] Step 2:

[0759] The user uploads not only text data but also media data such as images, audio, and video. Specifically, the user clicks the upload button, selects the required files from the file selection window, and completes the upload.

[0760] Step 3:

[0761] The terminal transmits the text data input by the user and the uploaded media data to the server. Specifically, the terminal creates an HTTP request and transmits the data to the server.

[0762] Step 4:

[0763] The server receives the data from the terminal and stores it in the database. Specifically, it analyzes the HTTP request, extracts the data, and executes the process to store it in the database.

[0764] Step 5:

[0765] The server analyzes the stored text data using natural language processing (NLP) technology, specifically using text analysis algorithms to extract and understand the key content of the goals and tasks.

[0766] Step 6:

[0767] The server analyzes the media data (images, audio, video) it receives. Specifically, it applies image recognition, audio analysis, and video analysis algorithms to classify and understand the content of the media data.

[0768] Step 7:

[0769] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.). For example, it can recognize nervousness from the user's facial expressions during a presentation.

[0770] Step 8:

[0771] The server integrates the results of the text data analysis, media data analysis, and emotion engine analysis to generate a specific coaching plan for the user to achieve their goals and solve their problems. Specifically, it creates feedback, advice, and a specific action plan in the form of text and media based on the analysis results.

[0772] Step 9:

[0773] The server sends the generated feedback and advice to the device. Specifically, the feedback data is sent as an HTTP response, and the device receives it.

[0774] Step 10:

[0775] The feedback received by the device is displayed to the user. Specifically, a UI (user interface) is constructed to display the feedback content on the screen in an appropriate format, allowing the user to easily access it.

[0776] Step 11:

[0777] The server periodically tracks the user's progress. Specifically, each time the user uploads new data (e.g., a new presentation video), it receives and analyzes it and evaluates the user's progress.

[0778] Step 12:

[0779] The user periodically uploads new data (e.g., a new presentation video). Specifically, the user repeatedly sends progress data from the device to the server.

[0780] Step 13:

[0781] The server compares the results of the analysis with the previous results and generates updated feedback, specifically, updates the feedback based on the new data and provides it to the user.

[0782] Step 14:

[0783] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state. Specifically, it selects the optimal feedback method based on the user's profile information and applies an individual approach.

[0784] This allows the system to provide users with efficient and personalized coaching to help them improve their careers and performance.

[0785] Example 2

[0786] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0787] Effective feedback and coaching are essential for modern businesspeople to achieve their goals and challenges. However, traditional systems struggle to provide personalized feedback in real time that reflects users' emotions and progress. Furthermore, the lack of detailed progress tracking and updated feedback makes it difficult for users to continuously improve.

[0788] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for a user to input goals and tasks and upload text data and media data, means for receiving and saving the input data, means for analyzing the received data and generating feedback for the user to achieve their goals and solve their tasks, means for recognizing the user's emotions from their facial expressions, voice, and body movements and adjusting the feedback based on the results, and means for providing the generated feedback to the user. This makes it possible to provide personalized feedback that takes the user's emotional state into consideration, allowing the user to efficiently continue making improvements toward achieving their goals.

[0789] A "user" is an individual or organization that uses the system to achieve a goal or task.

[0790] A "goal" is a specific outcome or objective that a user wishes to achieve.

[0791] A "challenge" is a problem or obstacle that a user must overcome.

[0792] "Text data" refers to information or content expressed in a written format.

[0793] "Media data" refers to visual or audible information such as images, audio, and video, and includes non-text data.

[0794] "Upload" refers to sending data from the user's terminal to the server.

[0795] A "server" is a central system that receives, stores, and analyzes input data, and generates and provides feedback to users.

[0796] The "emotion engine" is a system that recognizes a user's emotional state by analyzing their facial expressions, voice, and body movements.

[0797] "Feedback" refers to advice and evaluations to help users achieve their goals and solve their problems.

[0798] "Tracking" is the act of continuously monitoring and recording a user's progress and analyzing that data.

[0799] "Personalization" means tailoring feedback content and communication style based on the individual personality and preferences of each user.

[0800] MODE FOR CARRYING OUT THE INVENTION

[0801] System Program Overview

[0802] This invention is a system that provides effective feedback and coaching to businesspeople to help them achieve their goals and challenges. Furthermore, by combining it with an emotion engine, the system adjusts the feedback based on the user's emotions. The following specific hardware and software are used to implement this system:

[0803] Hardware and software used

[0804] 1. On the user's device:

[0805] PC, smartphone, etc. Users can input their goals and tasks and upload text and media data.

[0806] 2. Server:

[0807] It acts as a central system, receiving, storing, and analyzing input data and providing feedback to users.

[0808] 3. Emotion Engine:

[0809] A system that recognizes emotions from a user's facial expressions, tone of voice, body movements, etc. For example, it uses facial expression recognition technology and voice analysis technology.

[0810] Program processing overview

[0811] Entering user information and initial settings

[0812] A user uses a terminal to input basic information about themselves (such as name, occupation, goals, and tasks). This input includes not only text data, but also image, audio, and video data as needed. For example, a user who aims to succeed in a new project may set improving their presentation skills as a task and upload videos of past presentations to the system.

[0813] Sending and Receiving Data

[0814] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[0815] Data analysis

[0816] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies, including natural language processing (NLP) of text data, as well as image recognition, speech analysis, and video analysis. For example, a presentation video can be analyzed to generate feedback on pronunciation and the flow of the presentation.

[0817] Emotion analysis

[0818] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.). For example, it can recognize nervousness from the user's facial expressions during a presentation.

[0819] Generate a coaching plan

[0820] The server generates a specific coaching plan for the user based on the analysis results and the emotion engine results. This plan includes feedback, advice, and a specific action plan. For example, a training plan for improving presentation skills could be created, with specific points to improve and practice methods.

[0821] Providing Feedback

[0822] The server generates feedback and advice, which is then sent to the device and provided to the user. Feedback is provided in the form of text, audio, video, etc., and the user can review and use it as appropriate. The user can also receive personalized feedback based on their emotional state.

[0823] Tracking progress

[0824] The server periodically tracks the user's progress: the user periodically uploads new data (e.g. a new presentation video), which the server analyzes and provides up-to-date feedback.

[0825] Personalized and adaptive

[0826] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state, resulting in more effective coaching. For example, if the user is nervous, the server can provide advice on how to relax.

[0827] Specific examples and examples of input prompts to the generative AI model

[0828] Specific examples

[0829] 1. A user aims to improve their presentation skills in order to succeed in a new project.

[0830] 2. The user uploads past presentation videos to the system from their device.

[0831] 3. The server receives the video and begins analyzing it, generating feedback using NLP, image recognition, and voice analysis technologies.

[0832] 4. The emotion engine recognizes the user's state of tension from the video and sends the emotional state to the server.

[0833] 5. The server generates feedback and gives specific advice, such as "Practice speaking slowly to improve your pronunciation. Take deep breaths when you feel nervous."

[0834] 6. When the user uploads a new video, feedback will be provided again.

[0835] Example of input prompt for generative AI model

[0836] Analyze the following data:

[0837] Name: Yamada Taro

[0838] Job type: Sales

[0839] Goal: New project success

[0840] Challenge: Improve presentation skills

[0841] Uploaded data: Past presentation videos

[0842] Use this data to generate feedback and a specific coaching plan, including advice on what to do if you're nervous.

[0843] This system allows users to receive highly efficient, personalized coaching at low cost, effectively improving their careers and performance.

[0844] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0845] Step 1: Entering user information and initial settings

[0846] A user logs into the system using a terminal and enters their basic information (name, job title, goals, challenges, etc.). This input includes not only text data, but also image, audio, video, and other data as needed. For example, a user whose goal is to succeed in a new project may set improving their presentation skills as a challenge and upload videos of past presentations to the system.

[0847] Input: User basic information, text data, media data

[0848] Output: Save input data

[0849] Specific behavior:

[0850] A user accesses the system's login page and logs in by entering authentication information (user name and password).

[0851] After logging in, a basic information input form is displayed and the user enters the required data.

[0852] Enter your goals and objectives, and upload media data (such as presentation videos).

[0853] Step 2: Sending and Receiving Data

[0854] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[0855] Input: Data entered by the user

[0856] Output: Save data to database

[0857] Specific behavior:

[0858] When the user clicks the "Send" button, the terminal sends the data to the server.

[0859] The server processes the received data in real time and stores it in a database.

[0860] Step 3: Data analysis

[0861] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies such as image recognition, voice analysis, and video analysis, as well as natural language processing (NLP) of text data.

[0862] Input: User text data, media data

[0863] Output: Feedback generation

[0864] Specific behavior:

[0865] The server launches an NLP engine to analyze the meaning of the text data.

[0866] The video analysis engine analyzes the uploaded presentation video frame by frame and evaluates the user's speaking speed and clarity.

[0867] The voice analysis engine analyzes the tone and pace of the user's voice.

[0868] Step 4: Sentiment Analysis

[0869] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.).

[0870] Input: User's video data, audio data

[0871] Output: Emotion recognition result

[0872] Specific behavior:

[0873] The emotion engine uses facial expression recognition technology to analyze the user's facial features in the video.

[0874] The server receives the emotion analysis results and uses them to generate feedback.

[0875] Step 5: Create a coaching plan

[0876] The server generates a specific coaching plan for the user based on the results of the data analysis and the sentiment analysis, which includes feedback, advice, and a specific action plan.

[0877] Input: Data analysis results, emotion recognition results

[0878] Output: Coaching plan generated

[0879] Specific behavior:

[0880] The server generates detailed feedback on how to improve and practice your presentation.

[0881] The server will also provide relaxation techniques and advice to help relieve tension.

[0882] Step 6: Provide feedback

[0883] The server generates feedback and advice, which is sent to the device and provided to the user in the form of text, audio, video, etc.

[0884] Input: Generated feedback

[0885] Output: Providing feedback

[0886] Specific behavior:

[0887] The server generates feedback and sends it to the user's device as a push notification.

[0888] The feedback content is displayed on the terminal screen and the user can check it.

[0889] Step 7: Track your progress

[0890] The server periodically tracks the user's progress: the user periodically uploads new data (e.g. a new presentation video), which the server re-analyzes and provides updated feedback.

[0891] Input: User's new data

[0892] Output: Updated feedback

[0893] Specific behavior:

[0894] The server compares the user's past feedback with their current progress and generates an updated report.

[0895] When a user uploads a new presentation video, the server compares it with previous analysis results and identifies areas for improvement.

[0896] Step 8: Personalize and adapt

[0897] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state, resulting in more effective coaching.

[0898] Input: User's personality profile, emotion recognition results

[0899] Output: Personalized feedback

[0900] Specific behavior:

[0901] The server adjusts the feedback content based on the user's personality profile and past feedback results.

[0902] Providing feedback based on emotional state (e.g., suggesting ways to relax if you're feeling tense).

[0903] (Application example 2)

[0904] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0905] There is a need for systems that provide feedback and coaching to improve the work efficiency of robotic workers in factories, reduce errors, and maintain optimal work conditions. However, conventional systems lack sufficient analysis of motion and sensor data, and are unable to provide appropriate feedback based on emotional states. This results in poor robot performance and has a negative impact on work efficiency and quality.

[0906] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0907] In this invention, the server includes means for a user to input goals and tasks and upload text data and media data, means for receiving and saving the input data, means for analyzing the received data and generating feedback for the user to achieve their goals and solve their tasks, means for providing the generated feedback to the user, means for analyzing video of the robot's work and sensor data and generating feedback for improving work efficiency and reducing errors, and means for performing emotional analysis based on the robot's sensor data and providing appropriate feedback according to the robot's emotional state. This makes it possible to analyze the robot's operating status in real time and provide optimal feedback and coaching.

[0908] A "user" is someone who uses the system or provides data to help the robotic worker achieve its goals and solve its problems.

[0909] "Goal" refers to the result or purpose that a user aims to achieve.

[0910] A "challenge" refers to a problem or obstacle that a user must overcome in the process of achieving a goal.

[0911] "Text data" refers to data that includes character information entered by the user.

[0912] "Media data" refers to data in various formats such as images, audio, and video.

[0913] "Reception" refers to the process in which the server acquires data sent from the user terminal.

[0914] "Storage" refers to recording the received data in a database or storage within the server.

[0915] "Analysis" is the process of extracting and evaluating information based on the received data to help the user achieve their goals and solve their problems.

[0916] "Feedback" refers to information or advice returned to the user based on the analysis results.

[0917] "Robot work video" is video data that records the movements of robots operating in a factory.

[0918] "Sensor data" refers to data such as temperature, vibration, and position obtained from sensors installed on the robot.

[0919] "Emotion analysis" is the process of inferring a robot's emotional state based on its movements and sensor data.

[0920] "Appropriate feedback" refers to the most effective advice and information based on the robot's state, taking into account the results of emotion analysis.

[0921] This invention is a system for improving the work efficiency and reducing errors of robot workers in factories. The system includes means for a user to input goals and tasks and upload text data and media data, means for receiving and saving the data, means for analyzing the received data, means for providing generated feedback to the user, means for analyzing video of the robot's work and sensor data to generate feedback for improving work efficiency and reducing errors, and means for performing emotional analysis and providing feedback according to the emotional state.

[0922] The server receives data input from the user's device (for example, information about the goals and challenges of robot workers in a factory, along with the associated text and media data) and stores it in a database. Software libraries such as TensorFlow and OpenCV are used to analyze the received data. These libraries are used to analyze motion in videos and sensor data, and generate the feedback necessary to help the user achieve their goals and solve their problems.

[0923] Furthermore, the server uses an emotion analysis engine to estimate the robot's emotional state (e.g., stress or fatigue) using video and sensor data from the robot's work. The emotion analysis engine evaluates the robot's movements and sensor data based on a specific emotion model and adjusts the feedback content. The generated feedback may include specific improvements, training modules, or maintenance schedules to improve work efficiency.

[0924] As a concrete example of this system, imagine a robot working on a conveyor line. Videos of the robot's work are recorded and sensor data (such as vibration and temperature data) is acquired. The server analyzes this data and provides feedback suggesting techniques to optimize the robot's movements, or, if high stress is detected from the sensor data, feedback suggesting when maintenance should be performed.

[0925] An example prompt might look like this:

[0926] "What operational improvements are needed to ensure that this conveyor line robot operates efficiently? Also, what maintenance is appropriate if it is experiencing fatigue or stress?"

[0927] This allows robotic workers in factories to receive optimal feedback and coaching in real time, improving work efficiency and reducing errors.

[0928] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0929] Step 1:

[0930] The server provides a means for users to input goals and tasks and upload text and media data. This allows users to set specific goals and tasks for the system and send data related to them. Input data could be, for example, video of robots working in a factory or sensor data. This input data is sent to the server and stored.

[0931] Step 2:

[0932] The server has a means for receiving and storing the input data. The received data includes text data (goals and tasks), video data (videos of the robot working), and sensor data (temperature, vibration, etc.). The received data is recorded in a database for subsequent analysis.

[0933] Step 3:

[0934] The server analyzes the received data. Specifically, it uses TensorFlow and OpenCV to analyze the movements in the video and extract important movement information from each frame. It also analyzes sensor data to determine the robot's operating status and detect abnormalities. This allows it to identify, for example, areas where the robot's movements are lagging or where abnormal vibrations are occurring.

[0935] Step 4:

[0936] The server provides the generated feedback to the user. Based on the analysis results, it generates specific advice on operation optimization techniques and areas for improvement. For example, if a specific operation is slow, it will suggest techniques to optimize that operation, and if an abnormality is detected, it will suggest maintenance.

[0937] Step 5:

[0938] The server performs emotion analysis based on the robot's work video and sensor data. The emotion analysis engine evaluates the video and sensor data to estimate the robot's emotional state (such as stress or fatigue). This allows it to provide individual feedback based on its condition.

[0939] Step 6:

[0940] The server then provides final feedback to the user. Taking into account the results of sentiment analysis, the server provides more personalized feedback to the user. For example, if high stress is detected, the server provides feedback recommending regular rest and maintenance.

[0941] Step 7:

[0942] Users receive feedback provided by the server and use it to improve the robot's work and for maintenance. This improves the robot's work efficiency and reduces errors. For example, by implementing the proposed motion optimization technology, the speed and accuracy of work can be improved. In addition, by performing regular maintenance, the robot's uptime can be maximized.

[0943] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0944] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0945] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0946] [Third embodiment]

[0947] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0948] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0949] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0950] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0951] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0952] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0953] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0954] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0955] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0956] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0957] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0958] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0959] The present invention is a system that provides effective feedback and coaching to business people to help them achieve their goals and challenges. Specific embodiments for implementing the system will be described below.

[0960] System configuration

[0961] The system mainly includes the following components:

[0962] 1. User's device

[0963] A device (such as a computer or smartphone) on which users can enter their goals and challenges and upload text and media data.

[0964] 2. Server

[0965] A central system that receives, stores, analyzes input data, and provides feedback to users.

[0966] Program processing overview

[0967] 1. Entering user information and initial settings

[0968] A user uses a terminal to input their basic information (such as name, occupation, goals, and tasks). This input includes not only text data, but also image, audio, and video data as needed. For example, a user aiming to succeed in a new project may set improving their presentation skills as a task and upload videos of past presentations to the system.

[0969] 2. Sending and Receiving Data

[0970] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[0971] 3. Data Analysis

[0972] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies, including natural language processing (NLP) of text data, as well as image recognition, speech analysis, and video analysis. For example, a presentation video can be analyzed to generate feedback on pronunciation and the flow of the presentation.

[0973] 4. Create a coaching plan

[0974] Based on the analysis results, the server generates a specific coaching plan for the user, which includes feedback, advice, and a concrete action plan. For example, a training plan for improving presentation skills could be created, with specific points to improve and practice methods.

[0975] 5. Providing Feedback

[0976] The server then sends the generated feedback and advice to the terminal and provides it to the user. The feedback is provided in the form of text, audio, video, etc., and the user can check and use it as appropriate.

[0977] 6. Track your progress

[0978] The server periodically tracks the user's progress: the user periodically uploads new data (e.g., a new presentation video), which the server analyzes and provides up-to-date feedback.

[0979] 7. Personalization and Adaptation

[0980] The server personalizes the feedback and communication style based on the user's personality and preferences, resulting in more effective coaching. For example, it adopts a positive feedback style to enhance motivation.

[0981] As described above, the present invention realizes a system that provides users with inexpensive, highly accurate coaching that is available 24 hours a day, 365 days a year, allowing users to effectively improve their careers and performance.

[0982] The processing flow will be explained below.

[0983] Step 1:

[0984] The user uses the terminal to enter their basic information (name, occupation, goals, tasks, etc.). Specifically, the user enters the information into the input form on the terminal and sends the data by pressing the send button.

[0985] Step 2:

[0986] The user uploads not only text data but also media data such as images, audio, and video. Specifically, the user clicks the upload button, selects the required files from the file selection window, and completes the upload.

[0987] Step 3:

[0988] The terminal transmits the text data input by the user and the uploaded media data to the server. Specifically, the terminal creates an HTTP request and transmits the data to the server.

[0989] Step 4:

[0990] The server receives the data from the terminal and stores it in the database. Specifically, it analyzes the HTTP request, extracts the data, and executes the process to store it in the database.

[0991] Step 5:

[0992] The server analyzes the stored text data using natural language processing (NLP) technology, specifically using text analysis algorithms to extract and understand the key content of the goals and tasks.

[0993] Step 6:

[0994] The server analyzes the media data (images, audio, video) it receives. Specifically, it applies image recognition, audio analysis, and video analysis algorithms to classify and understand the content of the media data.

[0995] Step 7:

[0996] The server generates feedback based on the analysis results to help users achieve their goals and solve their problems. Specifically, it creates specific action plans and advice in the form of text or other media based on the analysis results.

[0997] Step 8:

[0998] The server sends the generated feedback and advice to the device. Specifically, the feedback data is sent as an HTTP response, and the device receives it.

[0999] Step 9:

[1000] The feedback received by the device is displayed to the user. Specifically, a UI (user interface) is constructed to display the feedback content on the screen in an appropriate format, allowing the user to easily access it.

[1001] Step 10:

[1002] The server periodically tracks the user's progress: each time the user uploads new data, it receives and analyzes it and evaluates the progress.

[1003] Step 11:

[1004] The user periodically uploads new data (e.g., a new presentation video). Specifically, the user repeatedly sends progress data from the device to the server.

[1005] Step 12:

[1006] The server compares the results of the analysis with the previous results and generates updated feedback, specifically, updates the feedback based on the new data and provides it to the user.

[1007] Step 13:

[1008] The server personalizes the feedback content and communication style based on the user's personality and preferences. Specifically, it selects the optimal feedback method based on the user's profile information and applies an individual approach.

[1009] This allows the system to provide users with efficient and personalized coaching, helping them improve their careers and performance.

[1010] Example 1

[1011] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1012] Conventional coaching systems require a lot of effort for users to input their goals and challenges, and provide general, blanket feedback that fails to adequately address individual needs. Furthermore, they lack a way to continuously track a user's progress or to personalize feedback based on personality and preferences. This makes it difficult to effectively support users in achieving their goals and resolving challenges.

[1013] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1014] In this invention, the server includes means for allowing a user to input goals and tasks and upload text data and media data, means for receiving and storing the input data, means for analyzing the received data and using natural language processing, image recognition, voice analysis, and video analysis to generate feedback for the user to achieve their goals and solve their tasks, and means for providing the generated feedback to the user, thereby enabling the provision of feedback that meets the individual needs of the user, continuous tracking of progress, and even personalized feedback.

[1015] A "user" is a person or organization that uses the system to input their goals and challenges and receive feedback.

[1016] A "goal" indicates the purpose or intention that a user is trying to achieve, and is information that is input into the system.

[1017] "Issues" indicate problems that users are trying to solve or areas that need improvement, and are information that is entered into the system.

[1018] "Text data" refers to text information entered by the user, including details of goals and tasks.

[1019] "Media data" refers to digital data such as images, audio, and video uploaded by users.

[1020] "Server" refers to the central system that receives, stores, analyzes, and generates and provides feedback on input data.

[1021] "Natural language processing" is a technique used by systems to analyze text data in order to understand the user's intent and meaning.

[1022] "Image recognition" is a technology that allows a system to analyze image data and understand its contents.

[1023] "Speech analysis" is a technology that allows a system to analyze voice data and evaluate its content and quality.

[1024] "Video analysis" is a technology that allows a system to analyze video data and evaluate its content and performance.

[1025] "Feedback" refers to information that provides evaluation and advice on the goals and tasks entered by the user.

[1026] "Personalization" refers to individually tailoring feedback content and communication style based on a user's personality and preferences.

[1027] "Tracking" means that the system continuously monitors and evaluates the user's progress.

[1028] The present invention is a system that provides effective feedback and coaching to businesspeople to help them achieve their goals and challenges. The system mainly includes the following components:

[1029] 1. User's device

[1030] Users use devices such as personal computers or smartphones to input goals and tasks as text data and upload media data such as images, audio, and video. This device acts as the user's interface, inputting data and receiving feedback.

[1031] Examples:

[1032] The user uses the smartphone app to enter their name (e.g., "Yamada Taro"), job title (e.g., "Sales Manager"), goal (e.g., "Success in a new project"), and task (e.g., "Improve presentation skills"), then presses the "Settings Complete" button. Next, they use the smartphone's camera function to record a video of a past presentation and upload it from the app.

[1033] 2. Server

[1034] The server receives text and media data entered from the user's device and stores it in a database. This data is analyzed using an analysis module within the server. Multimodal AI technologies such as natural language processing (NLP), image recognition, audio analysis, and video analysis are used for the analysis. Specific technologies used include Google Cloud Natural Language API and Google Cloud Video Intelligence API.

[1035] Examples:

[1036] The server analyzes the received text information and presentation video using the Google Cloud Natural Language API and Google Cloud Video Intelligence API. The server understands the user's intent and challenges from the text information, and evaluates pronunciation clarity, gestures, eye movements, etc. from the video data. The analysis results are stored in a dedicated table in the database.

[1037] 3. Providing Feedback

[1038] Based on the analysis results, the server generates a specific coaching plan for the user. The generated feedback and advice is sent to the user's device in text, audio, and video formats. The user can use this to improve their skills and solve problems.

[1039] Examples:

[1040] The server generates feedback data (e.g., "There were some parts where your pronunciation was unclear. Please try to speak more slowly.") and sends it to the user's smartphone. The user opens the app and checks the provided feedback.

[1041] The system based on the present invention is available to users 24 hours a day, 365 days a year, and provides inexpensive, highly accurate coaching, allowing users to effectively improve their careers and performance.

[1042] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1043] Step 1:

[1044] Entering user information and initial settings

[1045] Users start the application using their personal computer or smartphone, enter basic information (such as name, occupation, goals, and tasks), and upload image, audio, and video data as needed.

[1046] Input: Name, job title, goal, task, media data (images, audio, video)

[1047] Output: Packages the input data and converts it into a format that can be sent to the server.

[1048] Specific operation: The user opens the app on their smartphone and goes to the "Profile Settings" screen. They enter their name (e.g., "Yamada Taro"), job title (e.g., "Sales Manager"), goal (e.g., "Success in new projects"), and challenge (e.g., "Improve presentation skills") in the text boxes and press the "Settings Complete" button. Next, they record a video of a past presentation using the device's camera and upload it from within the app.

[1049] Step 2:

[1050] Sending and Receiving Data

[1051] The device compresses the input text and media data and sends it to the server via a secure protocol (e.g., HTTPS). The server receives this data and stores it in a dedicated database.

[1052] Input: Compressed text and media data

[1053] Output: User data stored in the server database

[1054] Specific operation: After the user presses the "Settings complete" button, the terminal compresses the data and sends it to the server using the SSL / TLS protocol. The server stores the received data in a database table dedicated to analysis.

[1055] Step 3:

[1056] Data analysis

[1057] The server uses a dedicated data analysis module to analyze the received data, using natural language processing (NLP), image recognition, audio analysis, and video analysis technologies.

[1058] Input: Stored text and media data

[1059] Output: Analysis results (data for creating feedback)

[1060] How it works: The server analyzes text information using the Google Cloud Natural Language API. It also analyzes uploaded presentation videos using the Google Cloud Video Intelligence API, evaluating pronunciation clarity, gestures, and eye movements. These results are stored in a database as analytical data that specifically indicates areas for improvement for the user.

[1061] Step 4:

[1062] Generate a coaching plan

[1063] Based on the analysis results, the server generates a specific coaching plan for the user to achieve their goals and solve their problems.

[1064] Input: Analysis result data

[1065] Output: Coaching plan (feedback, advice, specific action plan)

[1066] Specific operation: Based on the results of the analysis, the server uses the coaching plan generation module to create a specific action plan, such as "Practice speaking for 10 minutes every day to improve the fluency of your presentation." The generated plan is stored in a database table dedicated to user feedback.

[1067] Step 5:

[1068] Providing Feedback

[1069] The server sends the generated feedback and advice to the user's device, which the user can use to improve their skills and solve problems.

[1070] Input: Coaching plan data

[1071] Output: Feedback sent to the user device

[1072] Specific operation: The server generates feedback data (e.g., "There were some parts where your pronunciation was unclear. Please try to speak more slowly.") and sends it to the user's smartphone. The user opens the app and checks the provided feedback.

[1073] Step 6:

[1074] Tracking progress

[1075] The server continually tracks the user's progress and provides updated feedback as needed.

[1076] Input: Newly uploaded data (e.g., a new presentation video)

[1077] Output: Updated feedback

[1078] What it does: Users periodically upload new presentation videos from their devices. The server analyzes the videos, evaluates their progress, and provides new feedback.

[1079] Step 7:

[1080] Personalized and adaptive

[1081] The server personalizes the feedback and communication style based on the user's personality and preferences.

[1082] Input: Past feedback history and user response data

[1083] Output: Personalized feedback

[1084] Specific actions: The server analyzes the feedback history and the user's response patterns, generates personalized feedback such as "include positive encouragement" and "provide specific examples of success," and provides it to the user.

[1085] (Application example 1)

[1086] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1087] While factories need to improve work efficiency and accuracy, there is a lack of ways to coach individual workers in real time. It is also difficult to continuously track worker performance and provide individual feedback, making it difficult to maintain consistent work quality. Furthermore, while individualized instruction based on each worker's personality and preferences is required, current systems are nearly impossible to address.

[1088] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1089] In this invention, the server includes a means for users to input goals and tasks and upload text data and media data, a means for receiving and saving the input data, a means for analyzing the received data and generating feedback for users to achieve their goals and solve their tasks, and a means for recording factory workers' movements with a camera and providing feedback in real time. This allows for real-time coaching of factory workers' movements and the provision of individualized feedback. Furthermore, by continuously tracking the progress of workers and providing personalized guidance based on their individual personalities and preferences, it is possible to expect improved work quality and consistent performance.

[1090] A "user" is someone who uses this system to set goals and challenges and receives feedback and coaching based on those goals and challenges.

[1091] "Means for inputting goals and tasks" refers to a device or interface that allows a user to input their goals and current tasks to the system using text data or media data.

[1092] "Text data and media data" refers to digital data such as text information, images, audio, and video that users input or upload to the system.

[1093] "Means for receiving and storing" refers to a device or software that allows the system to receive text data and media data entered or uploaded by a user and store it in an appropriate location, such as a database.

[1094] "Means for analyzing and generating feedback" refers to devices or software that process received text data or media data using various analytical techniques and automatically generate feedback that helps the user achieve their goals or solve their problems.

[1095] The "means for providing feedback" refers to a device or software for transmitting the generated feedback to the user, and includes, for example, a smartphone or PC application.

[1096] "Means for recording the actions of factory workers" refers to devices or software that use cameras or sensors to photograph and record the actions of workers in a factory and send that data to the system.

[1097] "Means for providing real-time feedback" refers to a device or software that instantly analyzes recorded operational data of factory workers and provides immediate feedback.

[1098] A "means for continuous progress tracking" is a device or software that periodically tracks a user's or worker's achievements or work progress and generates new feedback based on that data.

[1099] "Personalization means" refers to a device or software that individually optimizes the content of feedback and communication style based on the user's personality and preferences.

[1100] This invention is a system for coaching factory workers' actions in real time to achieve efficient and accurate work. This system includes a series of processes in which users (workers) input their goals and tasks and receive feedback and coaching based on those goals and tasks.

[1101] Hardware and software used

[1102] 1. Hardware to be used

[1103] High-performance cameras: Record factory workers' movements in real time

[1104] Smart Glasses: Providing Visual Feedback to Workers

[1105] Factory robots: motion assistance and data collection

[1106] 2. Software to be used

[1107] Natural Language Processing (NLP) Library: SpaCy

[1108] Image recognition technology: TensorFlow + OpenCV

[1109] Server: Microsoft Azure

[1110] Performance analysis algorithm: Python + Scikit-learn

[1111] Program processing overview

[1112] 1. Data Collection:

[1113] The server collects real-time movement data of factory workers through high-performance cameras, including their work movements, speed, and accuracy.

[1114] 2. Data transmission:

[1115] The collected data is sent over a network to a server, which receives and securely stores the data.

[1116] 3. Data Analysis:

[1117] The server analyzes the received data using natural language processing with SpaCy and image recognition technology with TensorFlow and OpenCV. This analysis identifies areas that need improvement, such as work efficiency and accuracy.

[1118] 4. Coaching Plan Creation:

[1119] Based on the analysis results, the server uses Python and Scikit-learn to generate an individual coaching plan, which includes specific improvements and training methods to help workers achieve their goals efficiently.

[1120] 5. Providing Feedback:

[1121] The server provides the generated feedback to the worker in real time through smart glasses, which can also be provided as audio guidance or visual cues.

[1122] 6. Progress Tracking:

[1123] The server periodically tracks the worker's progress and updates the feedback as needed, allowing for sustained performance improvement.

[1124] Specific examples

[1125] For example, consider a factory worker performing an assembly task. A camera records the worker's movements in real time and sends the data to a server. The server analyzes the movement data and evaluates the worker's speed and frequency of mistakes. Based on the analysis results, the server generates a coaching plan that includes "points to improve work speed" and a "checklist for reducing mistakes." This feedback is provided to the worker in real time through the smart glasses, allowing the worker to immediately recognize areas for improvement and work more efficiently.

[1126] Prompt Sentence Examples

[1127] Analyze video and audio data of workers' movements and generate feedback to improve their performance. For example, evaluate their work speed, frequency of mistakes, and accuracy of work procedures, and create and provide a training plan based on that. Specifically, please advise on the following items:

[1128] 1. Measures to improve work speed

[1129] 2. Checkpoints to reduce mistakes

[1130] 3. Visual guides to help you follow the steps

[1131] This system is expected to significantly improve the efficiency and quality of factory workers' work, and by providing individual feedback to each worker, it will be possible to uniformly improve performance throughout the factory.

[1132] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1133] Program processing flow

[1134] Step 1:

[1135] Data collection

[1136] Actions: The terminal (factory robot) uses a high-performance camera to record the actions of factory workers in real time.

[1137] Input: Video and audio data of worker movements.

[1138] Output: Real-time recorded motion data.

[1139] How it works: Cameras track workers' movements and voices and record the data in storage.

[1140] Step 2:

[1141] Data transmission

[1142] Action: The device sends the recorded data to the server.

[1143] Input: Real-time recorded motion data.

[1144] Output: The operation data sent to the server.

[1145] Specific operation: Using the terminal's network module, the operation data is transferred to the specified directory on the server.

[1146] Step 3:

[1147] Data Receipt and Storage

[1148] Action: The server stores the received data.

[1149] Input: Operational data sent from the device.

[1150] Output: Saved operating data.

[1151] Specific operation: The server stores the received data in a database and prepares it for further analysis.

[1152] Step 4:

[1153] Data analysis

[1154] How it works: The server analyzes the stored data.

[1155] Input: Stored operating data.

[1156] Output: Analysis results (e.g., work speed, frequency of mistakes, accuracy assessment).

[1157] Specific behavior:

[1158] The server analyzes the motion data using TensorFlow + OpenCV.

[1159] Analyze the audio data using SpaCy.

[1160] Generate data-driven feedback using Scikit-learn models.

[1161] Step 5:

[1162] Coaching plan creation

[1163] How it works: The server generates a coaching plan based on the analysis results.

[1164] Input: Analysis results.

[1165] Output: Coaching plan.

[1166] Specific behavior:

[1167] Based on the analysis results, an individual coaching plan is created using a Python script.

[1168] The written coaching plan is saved in the database.

[1169] Step 6:

[1170] Providing Feedback

[1171] Action: The server provides the generated feedback to the user (worker).

[1172] Enter: Coaching Plan.

[1173] Output: Feedback provided to the user (worker).

[1174] Specific behavior:

[1175] The smart glasses provide real-time audio guidance and visual feedback to the user.

[1176] Feedback is projected directly into the user's field of vision, encouraging immediate improvement.

[1177] Step 7:

[1178] Progress Tracking

[1179] How it works: The server tracks the progress of the workers.

[1180] Input: New behavioral data collected periodically.

[1181] Output: Updated analysis results and feedback.

[1182] Specific behavior:

[1183] The server periodically receives and analyzes new data and updates the feedback based on the results.

[1184] Progress reports are generated and provided to users and administrators.

[1185] This process provides real-time feedback and continuous improvement to factory workers, and the personalized feedback is expected to improve overall efficiency and quality.

[1186] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1187] The present invention is a system that provides effective feedback and coaching to business people to help them achieve their goals and challenges, and further adjusts the feedback based on the user's emotions by combining it with an emotion engine. Specific embodiments for implementing the system are described below.

[1188] System configuration

[1189] The system mainly includes the following components:

[1190] 1. User's device

[1191] A device (such as a computer or smartphone) on which users can enter their goals and challenges and upload text and media data.

[1192] 2. Server

[1193] A central system that receives, stores, analyzes input data, and provides feedback to users.

[1194] 3. Emotion Engine

[1195] A system that recognizes emotions from the user's facial expressions, tone of voice, body movements, etc.

[1196] Program processing overview

[1197] 1. Entering user information and initial settings

[1198] A user uses a terminal to input basic information about themselves (such as name, occupation, goals, and tasks). This input includes not only text data, but also image, audio, and video data as needed. For example, a user who aims to succeed in a new project may set improving their presentation skills as a task and upload videos of past presentations to the system.

[1199] 2. Sending and Receiving Data

[1200] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[1201] 3. Data Analysis

[1202] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies, including natural language processing (NLP) of text data, as well as image recognition, speech analysis, and video analysis. For example, a presentation video can be analyzed to generate feedback on pronunciation and the flow of the presentation.

[1203] 4. Emotion analysis

[1204] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.). For example, it can recognize nervousness from the user's facial expressions during a presentation.

[1205] 5. Create a coaching plan

[1206] The server generates a specific coaching plan for the user based on the analysis results and the emotion engine results. This plan includes feedback, advice, and a specific action plan. For example, a training plan for improving presentation skills may be created, with specific points to improve and practice methods indicated. Advice based on the user's emotional state may also be included.

[1207] 6. Providing Feedback

[1208] The server generates feedback and advice, which is then sent to the device and provided to the user. The feedback is provided in the form of text, audio, video, etc., and the user can review and use it as appropriate. The user can also receive personalized feedback based on their emotional state.

[1209] 7. Track your progress

[1210] The server periodically tracks the user's progress: the user periodically uploads new data (e.g., a new presentation video), which the server analyzes and provides up-to-date feedback.

[1211] 8. Personalization and Adaptation

[1212] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state, resulting in more effective coaching. For example, if the user is nervous, the server can provide advice on how to relax.

[1213] This system allows users to receive highly efficient, personalized coaching at low cost, effectively improving their careers and performance.

[1214] The processing flow will be explained below.

[1215] Step 1:

[1216] The user uses the terminal to enter their basic information (name, occupation, goals, tasks, etc.). Specifically, the user enters the information into the input form on the terminal and sends the data by pressing the send button.

[1217] Step 2:

[1218] The user uploads not only text data but also media data such as images, audio, and video. Specifically, the user clicks the upload button, selects the required files from the file selection window, and completes the upload.

[1219] Step 3:

[1220] The terminal transmits the text data input by the user and the uploaded media data to the server. Specifically, the terminal creates an HTTP request and transmits the data to the server.

[1221] Step 4:

[1222] The server receives the data from the terminal and stores it in the database. Specifically, it analyzes the HTTP request, extracts the data, and executes the process to store it in the database.

[1223] Step 5:

[1224] The server analyzes the stored text data using natural language processing (NLP) technology, specifically using text analysis algorithms to extract and understand the key content of the goals and tasks.

[1225] Step 6:

[1226] The server analyzes the media data (images, audio, video) it receives. Specifically, it applies image recognition, audio analysis, and video analysis algorithms to classify and understand the content of the media data.

[1227] Step 7:

[1228] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.). For example, it can recognize nervousness from the user's facial expressions during a presentation.

[1229] Step 8:

[1230] The server integrates the results of the text data analysis, media data analysis, and emotion engine analysis to generate a specific coaching plan for the user to achieve their goals and solve their problems. Specifically, it creates feedback, advice, and a specific action plan in the form of text and media based on the analysis results.

[1231] Step 9:

[1232] The server sends the generated feedback and advice to the device. Specifically, the feedback data is sent as an HTTP response, and the device receives it.

[1233] Step 10:

[1234] The feedback received by the device is displayed to the user. Specifically, a UI (user interface) is constructed to display the feedback content on the screen in an appropriate format, allowing the user to easily access it.

[1235] Step 11:

[1236] The server periodically tracks the user's progress. Specifically, each time the user uploads new data (e.g., a new presentation video), it receives and analyzes it and evaluates the user's progress.

[1237] Step 12:

[1238] The user periodically uploads new data (e.g., a new presentation video). Specifically, the user repeatedly sends progress data from the device to the server.

[1239] Step 13:

[1240] The server compares the results of the analysis with the previous results and generates updated feedback, specifically, updates the feedback based on the new data and provides it to the user.

[1241] Step 14:

[1242] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state. Specifically, it selects the optimal feedback method based on the user's profile information and applies an individual approach.

[1243] This allows the system to provide users with efficient and personalized coaching to help them improve their careers and performance.

[1244] Example 2

[1245] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1246] Effective feedback and coaching are essential for modern businesspeople to achieve their goals and challenges. However, traditional systems struggle to provide personalized feedback in real time that reflects users' emotions and progress. Furthermore, the lack of detailed progress tracking and updated feedback makes it difficult for users to continuously improve.

[1247] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for a user to input goals and tasks and upload text data and media data, means for receiving and saving the input data, means for analyzing the received data and generating feedback for the user to achieve their goals and solve their tasks, means for recognizing the user's emotions from their facial expressions, voice, and body movements and adjusting the feedback based on the results, and means for providing the generated feedback to the user. This makes it possible to provide personalized feedback that takes the user's emotional state into consideration, allowing the user to efficiently continue making improvements toward achieving their goals.

[1248] A "user" is an individual or organization that uses the system to achieve a goal or task.

[1249] A "goal" is a specific outcome or objective that a user wishes to achieve.

[1250] A "challenge" is a problem or obstacle that a user must overcome.

[1251] "Text data" refers to information or content expressed in text format.

[1252] "Media data" refers to visual or audible information such as images, audio, and video, and includes non-text data.

[1253] "Upload" refers to sending data from the user's terminal to the server.

[1254] A "server" is a central system that receives, stores, and analyzes input data, and generates and provides feedback to users.

[1255] The "emotion engine" is a system that recognizes a user's emotional state by analyzing their facial expressions, voice, and body movements.

[1256] "Feedback" refers to advice and evaluations to help users achieve their goals and solve their problems.

[1257] "Tracking" is the act of continuously monitoring and recording a user's progress and analyzing that data.

[1258] "Personalization" means tailoring feedback content and communication style based on the individual personality and preferences of each user.

[1259] MODE FOR CARRYING OUT THE INVENTION

[1260] System Program Overview

[1261] This invention is a system that provides effective feedback and coaching to businesspeople to help them achieve their goals and challenges. Furthermore, by combining it with an emotion engine, the system adjusts the feedback based on the user's emotions. The following specific hardware and software are used to implement this system:

[1262] Hardware and software used

[1263] 1. On the user's device:

[1264] PC, smartphone, etc. Users can input their goals and tasks and upload text and media data.

[1265] 2. Server:

[1266] It acts as a central system, receiving, storing, and analyzing input data and providing feedback to users.

[1267] 3. Emotion Engine:

[1268] A system that recognizes emotions from a user's facial expressions, tone of voice, body movements, etc. For example, it uses facial expression recognition technology and voice analysis technology.

[1269] Program processing overview

[1270] Entering user information and initial settings

[1271] A user uses a terminal to input basic information about themselves (such as name, occupation, goals, and tasks). This input includes not only text data, but also image, audio, and video data as needed. For example, a user who aims to succeed in a new project may set improving their presentation skills as a task and upload videos of past presentations to the system.

[1272] Sending and Receiving Data

[1273] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[1274] Data analysis

[1275] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies, including natural language processing (NLP) of text data, as well as image recognition, speech analysis, and video analysis. For example, a presentation video can be analyzed to generate feedback on pronunciation and the flow of the presentation.

[1276] Emotion analysis

[1277] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.). For example, it can recognize nervousness from the user's facial expressions during a presentation.

[1278] Generate a coaching plan

[1279] The server generates a specific coaching plan for the user based on the analysis results and the emotion engine results. This plan includes feedback, advice, and a specific action plan. For example, a training plan for improving presentation skills could be created, with specific points to improve and practice methods.

[1280] Providing Feedback

[1281] The server generates feedback and advice, which is then sent to the device and provided to the user. Feedback is provided in the form of text, audio, video, etc., and the user can review and use it as appropriate. The user can also receive personalized feedback based on their emotional state.

[1282] Tracking progress

[1283] The server periodically tracks the user's progress: the user periodically uploads new data (e.g. a new presentation video), which the server analyzes and provides up-to-date feedback.

[1284] Personalized and adaptive

[1285] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state, resulting in more effective coaching. For example, if the user is nervous, the server can provide advice on how to relax.

[1286] Specific examples and examples of input prompts to the generative AI model

[1287] Specific examples

[1288] 1. A user aims to improve their presentation skills in order to succeed in a new project.

[1289] 2. The user uploads past presentation videos to the system from their device.

[1290] 3. The server receives the video and begins analyzing it, generating feedback using NLP, image recognition, and voice analysis technologies.

[1291] 4. The emotion engine recognizes the user's state of tension from the video and sends the emotional state to the server.

[1292] 5. The server generates feedback and gives specific advice, such as "Practice speaking slowly to improve your pronunciation. Take deep breaths when you feel nervous."

[1293] 6. When the user uploads a new video, feedback will be provided again.

[1294] Example of input prompt for generative AI model

[1295] Analyze the following data:

[1296] Name: Yamada Taro

[1297] Job type: Sales

[1298] Goal: New project success

[1299] Challenge: Improve presentation skills

[1300] Uploaded data: Past presentation videos

[1301] Use this data to generate feedback and a specific coaching plan, including advice on what to do if you're nervous.

[1302] This system allows users to receive highly efficient, personalized coaching at low cost, effectively improving their careers and performance.

[1303] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1304] Step 1: Entering user information and initial settings

[1305] A user logs into the system using a terminal and enters their basic information (name, job title, goals, challenges, etc.). This input includes not only text data, but also image, audio, video, and other data as needed. For example, a user whose goal is to succeed in a new project may set improving their presentation skills as a challenge and upload videos of past presentations to the system.

[1306] Input: User basic information, text data, media data

[1307] Output: Save input data

[1308] Specific behavior:

[1309] A user accesses the system's login page and logs in by entering authentication information (user name and password).

[1310] After logging in, a basic information input form is displayed and the user enters the required data.

[1311] Enter your goals and objectives, and upload media data (such as presentation videos).

[1312] Step 2: Sending and Receiving Data

[1313] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[1314] Input: Data entered by the user

[1315] Output: Save data to database

[1316] Specific behavior:

[1317] When the user clicks the "Send" button, the terminal sends the data to the server.

[1318] The server processes the received data in real time and stores it in a database.

[1319] Step 3: Data analysis

[1320] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies such as image recognition, voice analysis, and video analysis, as well as natural language processing (NLP) of text data.

[1321] Input: User text data, media data

[1322] Output: Feedback generation

[1323] Specific behavior:

[1324] The server launches an NLP engine to analyze the meaning of the text data.

[1325] The video analysis engine analyzes the uploaded presentation video frame by frame and evaluates the user's speaking speed and clarity.

[1326] The voice analysis engine analyzes the tone and pace of the user's voice.

[1327] Step 4: Sentiment Analysis

[1328] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.).

[1329] Input: User's video data, audio data

[1330] Output: Emotion recognition result

[1331] Specific behavior:

[1332] The emotion engine uses facial expression recognition technology to analyze the user's facial features in the video.

[1333] The server receives the emotion analysis results and uses them to generate feedback.

[1334] Step 5: Create a coaching plan

[1335] The server generates a specific coaching plan for the user based on the results of the data analysis and the sentiment analysis, which includes feedback, advice, and a specific action plan.

[1336] Input: Data analysis results, emotion recognition results

[1337] Output: Coaching plan generated

[1338] Specific behavior:

[1339] The server generates detailed feedback on how to improve and practice your presentation.

[1340] The server will also provide relaxation techniques and advice to help relieve tension.

[1341] Step 6: Provide feedback

[1342] The server generates feedback and advice, which is sent to the device and provided to the user in the form of text, audio, video, etc.

[1343] Input: Generated feedback

[1344] Output: Providing feedback

[1345] Specific behavior:

[1346] The server generates feedback and sends it to the user's device as a push notification.

[1347] The feedback content is displayed on the terminal screen and the user can check it.

[1348] Step 7: Track your progress

[1349] The server periodically tracks the user's progress: the user periodically uploads new data (e.g. a new presentation video), which the server re-analyzes and provides updated feedback.

[1350] Input: User's new data

[1351] Output: Updated feedback

[1352] Specific behavior:

[1353] The server compares the user's past feedback with their current progress and generates an updated report.

[1354] When a user uploads a new presentation video, the server compares it with previous analysis results and identifies areas for improvement.

[1355] Step 8: Personalize and adapt

[1356] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state, resulting in more effective coaching.

[1357] Input: User's personality profile, emotion recognition results

[1358] Output: Personalized feedback

[1359] Specific behavior:

[1360] The server adjusts the feedback content based on the user's personality profile and past feedback results.

[1361] Providing feedback based on emotional state (e.g., suggesting ways to relax if you're feeling tense).

[1362] (Application example 2)

[1363] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1364] There is a need for systems that provide feedback and coaching to improve the work efficiency of robotic workers in factories, reduce errors, and maintain optimal work conditions. However, conventional systems lack sufficient analysis of motion and sensor data, and are unable to provide appropriate feedback based on emotional states. This results in poor robot performance and has a negative impact on work efficiency and quality.

[1365] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1366] In this invention, the server includes means for a user to input goals and tasks and upload text data and media data, means for receiving and saving the input data, means for analyzing the received data and generating feedback for the user to achieve their goals and solve their tasks, means for providing the generated feedback to the user, means for analyzing video of the robot's work and sensor data and generating feedback for improving work efficiency and reducing errors, and means for performing emotional analysis based on the robot's sensor data and providing appropriate feedback according to the robot's emotional state. This makes it possible to analyze the robot's operating status in real time and provide optimal feedback and coaching.

[1367] A "user" is someone who uses the system or provides data to help the robotic worker achieve its goals and solve its problems.

[1368] "Goal" refers to the result or purpose that a user aims to achieve.

[1369] A "challenge" refers to a problem or obstacle that a user must overcome in the process of achieving a goal.

[1370] "Text data" refers to data that includes character information entered by the user.

[1371] "Media data" refers to data in various formats such as images, audio, and video.

[1372] "Reception" refers to the process in which the server acquires data sent from the user terminal.

[1373] "Storage" refers to recording the received data in a database or storage within the server.

[1374] "Analysis" is the process of extracting and evaluating information based on the received data to help the user achieve their goals and solve their problems.

[1375] "Feedback" refers to information or advice returned to the user based on the analysis results.

[1376] "Robot work video" is video data that records the movements of robots operating in a factory.

[1377] "Sensor data" refers to data such as temperature, vibration, and position obtained from sensors installed on the robot.

[1378] "Emotion analysis" is the process of inferring a robot's emotional state based on its movements and sensor data.

[1379] "Appropriate feedback" refers to the most effective advice and information based on the robot's state, taking into account the results of emotion analysis.

[1380] This invention is a system for improving the work efficiency and reducing errors of robot workers in factories. The system includes means for a user to input goals and tasks and upload text data and media data, means for receiving and saving the data, means for analyzing the received data, means for providing generated feedback to the user, means for analyzing video of the robot's work and sensor data to generate feedback for improving work efficiency and reducing errors, and means for performing emotional analysis and providing feedback according to the emotional state.

[1381] The server receives data input from the user's device (for example, information about the goals and challenges of robot workers in a factory, along with the associated text and media data) and stores it in a database. Software libraries such as TensorFlow and OpenCV are used to analyze the received data. These libraries are used to analyze motion in videos and sensor data, and generate the feedback necessary to help the user achieve their goals and solve their problems.

[1382] Furthermore, the server uses an emotion analysis engine to estimate the robot's emotional state (e.g., stress or fatigue) using video and sensor data from the robot's work. The emotion analysis engine evaluates the robot's movements and sensor data based on a specific emotion model and adjusts the feedback content. The generated feedback may include specific improvements, training modules, or maintenance schedules to improve work efficiency.

[1383] As a concrete example of this system, imagine a robot working on a conveyor line. Videos of the robot's work are recorded and sensor data (such as vibration and temperature data) is acquired. The server analyzes this data and provides feedback suggesting techniques to optimize the robot's movements, or, if high stress is detected from the sensor data, feedback suggesting when maintenance should be performed.

[1384] An example prompt might look like this:

[1385] "What operational improvements are needed to ensure that this conveyor line robot operates efficiently? Also, what maintenance is appropriate if it is experiencing fatigue or stress?"

[1386] This allows robotic workers in factories to receive optimal feedback and coaching in real time, improving work efficiency and reducing errors.

[1387] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1388] Step 1:

[1389] The server provides a means for users to input goals and tasks and upload text and media data. This allows users to set specific goals and tasks for the system and send data related to them. Input data could be, for example, video of robots working in a factory or sensor data. This input data is sent to the server and stored.

[1390] Step 2:

[1391] The server has a means for receiving and storing the input data. The received data includes text data (goals and tasks), video data (videos of the robot working), and sensor data (temperature, vibration, etc.). The received data is recorded in a database for subsequent analysis.

[1392] Step 3:

[1393] The server analyzes the received data. Specifically, it uses TensorFlow and OpenCV to analyze the movements in the video and extract important movement information from each frame. It also analyzes sensor data to determine the robot's operating status and detect abnormalities. This allows it to identify, for example, areas where the robot's movements are lagging or where abnormal vibrations are occurring.

[1394] Step 4:

[1395] The server provides the generated feedback to the user. Based on the analysis results, it generates specific advice on operation optimization techniques and areas for improvement. For example, if a specific operation is slow, it will suggest techniques to optimize that operation, and if an abnormality is detected, it will suggest maintenance.

[1396] Step 5:

[1397] The server performs emotion analysis based on the robot's work video and sensor data. The emotion analysis engine evaluates the video and sensor data to estimate the robot's emotional state (such as stress or fatigue). This allows it to provide individual feedback based on its condition.

[1398] Step 6:

[1399] The server then provides final feedback to the user. Taking into account the results of sentiment analysis, the server provides more personalized feedback to the user. For example, if high stress is detected, the server provides feedback recommending regular rest and maintenance.

[1400] Step 7:

[1401] Users receive feedback provided by the server and use it to improve the robot's work and for maintenance. This improves the robot's work efficiency and reduces errors. For example, by implementing the proposed motion optimization technology, the speed and accuracy of work can be improved. In addition, by performing regular maintenance, the robot's uptime can be maximized.

[1402] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1403] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1404] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1405] [Fourth embodiment]

[1406] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1407] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1408] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1409] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1410] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1411] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1412] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1413] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1414] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1415] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1416] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1417] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1418] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1419] The present invention is a system that provides effective feedback and coaching to business people to help them achieve their goals and challenges. Specific embodiments for implementing the system will be described below.

[1420] System configuration

[1421] The system mainly includes the following components:

[1422] 1. User's device

[1423] A device (such as a computer or smartphone) on which users can enter their goals and challenges and upload text and media data.

[1424] 2. Server

[1425] A central system that receives, stores, analyzes input data, and provides feedback to users.

[1426] Program processing overview

[1427] 1. Entering user information and initial settings

[1428] A user uses a terminal to input their basic information (such as name, occupation, goals, and tasks). This input includes not only text data, but also image, audio, and video data as needed. For example, a user aiming to succeed in a new project may set improving their presentation skills as a task and upload videos of past presentations to the system.

[1429] 2. Sending and Receiving Data

[1430] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[1431] 3. Data Analysis

[1432] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies, including natural language processing (NLP) of text data, as well as image recognition, speech analysis, and video analysis. For example, a presentation video can be analyzed to generate feedback on pronunciation and the flow of the presentation.

[1433] 4. Create a coaching plan

[1434] Based on the analysis results, the server generates a specific coaching plan for the user, which includes feedback, advice, and a concrete action plan. For example, a training plan for improving presentation skills could be created, with specific points to improve and practice methods.

[1435] 5. Providing Feedback

[1436] The server then sends the generated feedback and advice to the terminal and provides it to the user. The feedback is provided in the form of text, audio, video, etc., and the user can check and use it as appropriate.

[1437] 6. Track your progress

[1438] The server periodically tracks the user's progress: the user periodically uploads new data (e.g., a new presentation video), which the server analyzes and provides up-to-date feedback.

[1439] 7. Personalization and Adaptation

[1440] The server personalizes the feedback and communication style based on the user's personality and preferences, resulting in more effective coaching. For example, it adopts a positive feedback style to enhance motivation.

[1441] As described above, the present invention realizes a system that provides users with inexpensive, highly accurate coaching that is available 24 hours a day, 365 days a year, allowing users to effectively improve their careers and performance.

[1442] The processing flow will be explained below.

[1443] Step 1:

[1444] The user uses the terminal to enter their basic information (name, occupation, goals, tasks, etc.). Specifically, the user enters the information into the input form on the terminal and sends the data by pressing the send button.

[1445] Step 2:

[1446] The user uploads not only text data but also media data such as images, audio, and video. Specifically, the user clicks the upload button, selects the required files from the file selection window, and completes the upload.

[1447] Step 3:

[1448] The terminal transmits the text data input by the user and the uploaded media data to the server. Specifically, the terminal creates an HTTP request and transmits the data to the server.

[1449] Step 4:

[1450] The server receives the data from the terminal and stores it in the database. Specifically, it analyzes the HTTP request, extracts the data, and executes the process to store it in the database.

[1451] Step 5:

[1452] The server analyzes the stored text data using natural language processing (NLP) technology, specifically using text analysis algorithms to extract and understand the key content of the goals and tasks.

[1453] Step 6:

[1454] The server analyzes the media data (images, audio, video) it receives. Specifically, it applies image recognition, audio analysis, and video analysis algorithms to classify and understand the content of the media data.

[1455] Step 7:

[1456] The server generates feedback based on the analysis results to help users achieve their goals and solve their problems. Specifically, it creates specific action plans and advice in the form of text or other media based on the analysis results.

[1457] Step 8:

[1458] The server sends the generated feedback and advice to the device. Specifically, the feedback data is sent as an HTTP response, and the device receives it.

[1459] Step 9:

[1460] The feedback received by the device is displayed to the user. Specifically, a UI (user interface) is constructed to display the feedback content on the screen in an appropriate format, allowing the user to easily access it.

[1461] Step 10:

[1462] The server periodically tracks the user's progress: each time the user uploads new data, it receives and analyzes it and evaluates the progress.

[1463] Step 11:

[1464] The user periodically uploads new data (e.g., a new presentation video). Specifically, the user repeatedly sends progress data from the device to the server.

[1465] Step 12:

[1466] The server compares the results of the analysis with the previous results and generates updated feedback, specifically, updates the feedback based on the new data and provides it to the user.

[1467] Step 13:

[1468] The server personalizes the feedback content and communication style based on the user's personality and preferences. Specifically, it selects the optimal feedback method based on the user's profile information and applies an individual approach.

[1469] This allows the system to provide users with efficient and personalized coaching, helping them improve their careers and performance.

[1470] Example 1

[1471] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1472] Conventional coaching systems require a lot of effort for users to input their goals and challenges, and provide general, blanket feedback that fails to adequately address individual needs. Furthermore, they lack a way to continuously track a user's progress or to personalize feedback based on personality and preferences. This makes it difficult to effectively support users in achieving their goals and resolving challenges.

[1473] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1474] In this invention, the server includes means for allowing a user to input goals and tasks and upload text data and media data, means for receiving and storing the input data, means for analyzing the received data and using natural language processing, image recognition, voice analysis, and video analysis to generate feedback for the user to achieve their goals and solve their tasks, and means for providing the generated feedback to the user, thereby enabling the provision of feedback that meets the individual needs of the user, continuous tracking of progress, and even personalized feedback.

[1475] A "user" is a person or organization that uses the system to input their goals and challenges and receive feedback.

[1476] A "goal" indicates the purpose or intention that a user is trying to achieve, and is information that is input into the system.

[1477] "Issues" indicate problems that users are trying to solve or areas that need improvement, and are information that is entered into the system.

[1478] "Text data" refers to text information entered by the user, including details of goals and tasks.

[1479] "Media data" refers to digital data such as images, audio, and video uploaded by users.

[1480] "Server" refers to the central system that receives, stores, analyzes, and generates and provides feedback on input data.

[1481] "Natural language processing" is a technique used by systems to analyze text data in order to understand the user's intent and meaning.

[1482] "Image recognition" is a technology that allows a system to analyze image data and understand its contents.

[1483] "Speech analysis" is a technology that allows a system to analyze voice data and evaluate its content and quality.

[1484] "Video analysis" is a technology that allows a system to analyze video data and evaluate its content and performance.

[1485] "Feedback" refers to information that provides evaluation and advice on the goals and tasks entered by the user.

[1486] "Personalization" refers to individually tailoring feedback content and communication style based on a user's personality and preferences.

[1487] "Tracking" means that the system continuously monitors and evaluates the user's progress.

[1488] The present invention is a system that provides effective feedback and coaching to businesspeople to help them achieve their goals and challenges. The system mainly includes the following components:

[1489] 1. User's device

[1490] Users use devices such as personal computers or smartphones to input goals and tasks as text data and upload media data such as images, audio, and video. This device acts as the user's interface, inputting data and receiving feedback.

[1491] Examples:

[1492] The user uses the smartphone app to enter their name (e.g., "Yamada Taro"), job title (e.g., "Sales Manager"), goal (e.g., "Success in a new project"), and task (e.g., "Improve presentation skills"), then presses the "Settings Complete" button. Next, they use the smartphone's camera function to record a video of a past presentation and upload it from the app.

[1493] 2. Server

[1494] The server receives text and media data entered from the user's device and stores it in a database. This data is analyzed using an analysis module within the server. Multimodal AI technologies such as natural language processing (NLP), image recognition, audio analysis, and video analysis are used for the analysis. Specific technologies used include Google Cloud Natural Language API and Google Cloud Video Intelligence API.

[1495] Examples:

[1496] The server analyzes the received text information and presentation video using the Google Cloud Natural Language API and Google Cloud Video Intelligence API. The server understands the user's intent and challenges from the text information, and evaluates pronunciation clarity, gestures, eye movements, etc. from the video data. The analysis results are stored in a dedicated table in the database.

[1497] 3. Providing Feedback

[1498] Based on the analysis results, the server generates a specific coaching plan for the user. The generated feedback and advice is sent to the user's device in text, audio, and video formats. The user can use this to improve their skills and solve problems.

[1499] Examples:

[1500] The server generates feedback data (e.g., "There were some parts where your pronunciation was unclear. Please try to speak more slowly.") and sends it to the user's smartphone. The user opens the app and checks the provided feedback.

[1501] The system based on the present invention is available to users 24 hours a day, 365 days a year, and provides inexpensive, highly accurate coaching, allowing users to effectively improve their careers and performance.

[1502] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1503] Step 1:

[1504] Entering user information and initial settings

[1505] Users launch the application using their personal computer or smartphone, enter basic information (such as name, occupation, goals, and tasks), and upload image, audio, and video data as needed.

[1506] Input: Name, job title, goal, task, media data (images, audio, video)

[1507] Output: Packages the input data and converts it into a format that can be sent to the server.

[1508] Specific operation: The user opens the app on their smartphone and goes to the "Profile Settings" screen. They enter their name (e.g., "Taro Yamada"), job title (e.g., "Sales Manager"), goal (e.g., "Success in new projects"), and challenge (e.g., "Improve presentation skills") in the text boxes and press the "Settings Complete" button. Next, they record a video of a past presentation using the device's camera and upload it from within the app.

[1509] Step 2:

[1510] Sending and Receiving Data

[1511] The device compresses the input text and media data and sends it to the server via a secure protocol (e.g., HTTPS). The server receives this data and stores it in a dedicated database.

[1512] Input: Compressed text and media data

[1513] Output: User data stored in the server database

[1514] Specific operation: After the user presses the "Settings complete" button, the terminal compresses the data and sends it to the server using the SSL / TLS protocol. The server stores the received data in a database table dedicated to analysis.

[1515] Step 3:

[1516] Data analysis

[1517] The server uses a dedicated data analysis module to analyze the received data, using natural language processing (NLP), image recognition, audio analysis, and video analysis technologies.

[1518] Input: Stored text and media data

[1519] Output: Analysis results (data for creating feedback)

[1520] How it works: The server analyzes text information using the Google Cloud Natural Language API. It also analyzes uploaded presentation videos using the Google Cloud Video Intelligence API, evaluating pronunciation clarity, gestures, and eye movements. These results are stored in a database as analytical data that specifically indicates areas for improvement for the user.

[1521] Step 4:

[1522] Generate a coaching plan

[1523] Based on the analysis results, the server generates a specific coaching plan for the user to achieve their goals and solve their problems.

[1524] Input: Analysis result data

[1525] Output: Coaching plan (feedback, advice, specific action plan)

[1526] Specific operation: Based on the results of the analysis, the server uses the coaching plan generation module to create a specific action plan, such as "Practice speaking for 10 minutes every day to improve the fluency of your presentation." The generated plan is stored in a database table dedicated to user feedback.

[1527] Step 5:

[1528] Providing feedback

[1529] The server sends the generated feedback and advice to the user's device, which the user can use to improve their skills and solve problems.

[1530] Input: Coaching plan data

[1531] Output: Feedback sent to the user device

[1532] Specific operation: The server generates feedback data (e.g., "There were some parts where your pronunciation was unclear. Please try to speak more slowly.") and sends it to the user's smartphone. The user opens the app and checks the provided feedback.

[1533] Step 6:

[1534] Tracking progress

[1535] The server continually tracks the user's progress and provides updated feedback as needed.

[1536] Input: Newly uploaded data (e.g., a new presentation video)

[1537] Output: Updated feedback

[1538] What it does: Users periodically upload new presentation videos from their devices. The server analyzes the videos, evaluates their progress, and provides new feedback.

[1539] Step 7:

[1540] Personalized and adaptive

[1541] The server personalizes the feedback and communication style based on the user's personality and preferences.

[1542] Input: Past feedback history and user response data

[1543] Output: Personalized feedback

[1544] Specific actions: The server analyzes the feedback history and the user's response patterns, generates personalized feedback such as "include positive encouragement" and "provide specific examples of success," and provides it to the user.

[1545] (Application example 1)

[1546] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1547] While factories need to improve work efficiency and accuracy, there is a lack of ways to coach individual workers in real time. It is also difficult to continuously track worker performance and provide individual feedback, making it difficult to maintain consistent work quality. Furthermore, while individualized instruction based on each worker's personality and preferences is required, current systems are nearly impossible to address.

[1548] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1549] In this invention, the server includes a means for users to input goals and tasks and upload text data and media data, a means for receiving and saving the input data, a means for analyzing the received data and generating feedback for users to achieve their goals and solve their tasks, and a means for recording factory workers' movements with a camera and providing feedback in real time. This allows for real-time coaching of factory workers' movements and the provision of individualized feedback. Furthermore, by continuously tracking the progress of workers and providing personalized guidance based on their individual personalities and preferences, it is possible to expect improved work quality and consistent performance.

[1550] A "user" is someone who uses this system to set goals and challenges and receives feedback and coaching based on those goals and challenges.

[1551] "Means for inputting goals and tasks" refers to a device or interface that allows a user to input their goals and current tasks to the system using text data or media data.

[1552] "Text data and media data" refers to digital data such as text information, images, audio, and video that users input or upload to the system.

[1553] "Means for receiving and storing" refers to a device or software that allows the system to receive text data and media data entered or uploaded by a user and store it in an appropriate location, such as a database.

[1554] "Means for analyzing and generating feedback" refers to devices or software that process received text data or media data using various analytical techniques and automatically generate feedback that helps the user achieve their goals or solve their problems.

[1555] The "means for providing feedback" refers to a device or software for transmitting the generated feedback to the user, and includes, for example, a smartphone or PC application.

[1556] "Means for recording the actions of factory workers" refers to devices or software that use cameras or sensors to photograph and record the actions of workers in a factory and send that data to the system.

[1557] "Means for providing real-time feedback" refers to a device or software that instantly analyzes recorded operational data of factory workers and provides immediate feedback.

[1558] A "means for continuous progress tracking" is a device or software that periodically tracks a user's or worker's achievements or work progress and generates new feedback based on that data.

[1559] "Personalization means" refers to a device or software that individually optimizes the content of feedback and communication style based on the user's personality and preferences.

[1560] This invention is a system for coaching factory workers' actions in real time to achieve efficient and accurate work. This system includes a series of processes in which users (workers) input their goals and tasks and receive feedback and coaching based on those goals and tasks.

[1561] Hardware and software used

[1562] 1. Hardware to be used

[1563] High-performance cameras: Record factory workers' movements in real time

[1564] Smart Glasses: Providing Visual Feedback to Workers

[1565] Factory robots: motion assistance and data collection

[1566] 2. Software to be used

[1567] Natural Language Processing (NLP) Library: SpaCy

[1568] Image recognition technology: TensorFlow + OpenCV

[1569] Server: Microsoft Azure

[1570] Performance analysis algorithm: Python + Scikit-learn

[1571] Program processing overview

[1572] 1. Data Collection:

[1573] The server collects real-time movement data of factory workers through high-performance cameras, including their work movements, speed, and accuracy.

[1574] 2. Data transmission:

[1575] The collected data is sent over a network to a server, which receives and securely stores the data.

[1576] 3. Data Analysis:

[1577] The server analyzes the received data using natural language processing with SpaCy and image recognition technology with TensorFlow and OpenCV. This analysis identifies areas that need improvement, such as work efficiency and accuracy.

[1578] 4. Coaching Plan Creation:

[1579] Based on the analysis results, the server uses Python and Scikit-learn to generate an individual coaching plan, which includes specific improvements and training methods to help workers achieve their goals efficiently.

[1580] 5. Providing Feedback:

[1581] The server provides the generated feedback to the worker in real time through smart glasses, which can also be provided as audio guidance or visual cues.

[1582] 6. Progress Tracking:

[1583] The server periodically tracks the worker's progress and updates the feedback as needed, allowing for sustained performance improvement.

[1584] Specific examples

[1585] For example, consider a factory worker performing an assembly task. A camera records the worker's movements in real time and sends the data to a server. The server analyzes the movement data and evaluates the worker's speed and frequency of mistakes. Based on the analysis results, the server generates a coaching plan that includes "points to improve work speed" and a "checklist for reducing mistakes." This feedback is provided to the worker in real time through the smart glasses, allowing the worker to immediately recognize areas for improvement and work more efficiently.

[1586] Prompt Sentence Examples

[1587] Analyze video and audio data of workers' movements and generate feedback to improve their performance. For example, evaluate their work speed, frequency of mistakes, and accuracy of work procedures, and create and provide a training plan based on that. Specifically, please advise on the following items:

[1588] 1. Measures to improve work speed

[1589] 2. Checkpoints to reduce mistakes

[1590] 3. Visual guides to help you follow the steps

[1591] This system is expected to significantly improve the efficiency and quality of factory workers' work, and by providing individual feedback to each worker, it will be possible to uniformly improve performance throughout the factory.

[1592] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1593] Program processing flow

[1594] Step 1:

[1595] Data collection

[1596] Actions: The terminal (factory robot) uses a high-performance camera to record the actions of factory workers in real time.

[1597] Input: Video and audio data of worker movements.

[1598] Output: Real-time recorded motion data.

[1599] How it works: Cameras track workers' movements and voices and record the data in storage.

[1600] Step 2:

[1601] Data transmission

[1602] Action: The device sends the recorded data to the server.

[1603] Input: Real-time recorded motion data.

[1604] Output: The operation data sent to the server.

[1605] Specific operation: Using the terminal's network module, the operation data is transferred to the specified directory on the server.

[1606] Step 3:

[1607] Data Receipt and Storage

[1608] Action: The server stores the received data.

[1609] Input: Operational data sent from the device.

[1610] Output: Saved operating data.

[1611] Specific operation: The server stores the received data in a database and prepares it for further analysis.

[1612] Step 4:

[1613] Data analysis

[1614] How it works: The server analyzes the stored data.

[1615] Input: Stored operating data.

[1616] Output: Analysis results (e.g., work speed, frequency of mistakes, accuracy assessment).

[1617] Specific behavior:

[1618] The server analyzes the motion data using TensorFlow + OpenCV.

[1619] Analyze the audio data using SpaCy.

[1620] Generate data-driven feedback using Scikit-learn models.

[1621] Step 5:

[1622] Coaching plan creation

[1623] How it works: The server generates a coaching plan based on the analysis results.

[1624] Input: Analysis results.

[1625] Output: Coaching plan.

[1626] Specific behavior:

[1627] Based on the analysis results, an individual coaching plan is created using a Python script.

[1628] The written coaching plan is saved in the database.

[1629] Step 6:

[1630] Providing Feedback

[1631] Action: The server provides the generated feedback to the user (worker).

[1632] Enter: Coaching Plan.

[1633] Output: Feedback provided to the user (worker).

[1634] Specific behavior:

[1635] The smart glasses provide real-time audio guidance and visual feedback to the user.

[1636] Feedback is projected directly into the user's field of vision, encouraging immediate improvement.

[1637] Step 7:

[1638] Progress Tracking

[1639] How it works: The server tracks the progress of the workers.

[1640] Input: New behavioral data collected periodically.

[1641] Output: Updated analysis results and feedback.

[1642] Specific behavior:

[1643] The server periodically receives and analyzes new data and updates the feedback based on the results.

[1644] Progress reports are generated and provided to users and administrators.

[1645] This process provides real-time feedback and continuous improvement to factory workers, and the personalized feedback is expected to improve overall efficiency and quality.

[1646] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1647] The present invention is a system that provides effective feedback and coaching to business people to help them achieve their goals and challenges, and further adjusts the feedback based on the user's emotions by combining it with an emotion engine. Specific embodiments for implementing the system are described below.

[1648] System configuration

[1649] The system mainly includes the following components:

[1650] 1. User's device

[1651] A device (such as a computer or smartphone) on which users can enter their goals and challenges and upload text and media data.

[1652] 2. Server

[1653] A central system that receives, stores, analyzes input data, and provides feedback to users.

[1654] 3. Emotion Engine

[1655] A system that recognizes emotions from the user's facial expressions, tone of voice, body movements, etc.

[1656] Program processing overview

[1657] 1. Entering user information and initial settings

[1658] A user uses a terminal to input basic information about themselves (such as name, occupation, goals, and tasks). This input includes not only text data, but also image, audio, and video data as needed. For example, a user who aims to succeed in a new project may set improving their presentation skills as a task and upload videos of past presentations to the system.

[1659] 2. Sending and Receiving Data

[1660] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[1661] 3. Data Analysis

[1662] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies, including natural language processing (NLP) of text data, as well as image recognition, speech analysis, and video analysis. For example, a presentation video can be analyzed to generate feedback on pronunciation and the flow of the presentation.

[1663] 4. Emotion analysis

[1664] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.). For example, it can recognize nervousness from the user's facial expressions during a presentation.

[1665] 5. Create a coaching plan

[1666] The server generates a specific coaching plan for the user based on the analysis results and the emotion engine results. This plan includes feedback, advice, and a specific action plan. For example, a training plan for improving presentation skills may be created, with specific points to improve and practice methods indicated. Advice based on the user's emotional state may also be included.

[1667] 6. Providing Feedback

[1668] The server generates feedback and advice, which is then sent to the device and provided to the user. The feedback is provided in the form of text, audio, video, etc., and the user can review and use it as appropriate. The user can also receive personalized feedback based on their emotional state.

[1669] 7. Track your progress

[1670] The server periodically tracks the user's progress: the user periodically uploads new data (e.g., a new presentation video), which the server analyzes and provides up-to-date feedback.

[1671] 8. Personalization and Adaptation

[1672] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state, resulting in more effective coaching. For example, if the user is nervous, the server can provide advice on how to relax.

[1673] This system allows users to receive highly efficient, personalized coaching at low cost, effectively improving their careers and performance.

[1674] The processing flow will be explained below.

[1675] Step 1:

[1676] The user uses the terminal to enter their basic information (name, occupation, goals, tasks, etc.). Specifically, the user enters the information into the input form on the terminal and sends the data by pressing the send button.

[1677] Step 2:

[1678] The user uploads not only text data but also media data such as images, audio, and video. Specifically, the user clicks the upload button, selects the required files from the file selection window, and completes the upload.

[1679] Step 3:

[1680] The terminal transmits the text data input by the user and the uploaded media data to the server. Specifically, the terminal creates an HTTP request and transmits the data to the server.

[1681] Step 4:

[1682] The server receives the data from the terminal and stores it in the database. Specifically, it analyzes the HTTP request, extracts the data, and executes the process to store it in the database.

[1683] Step 5:

[1684] The server analyzes the stored text data using natural language processing (NLP) technology, specifically using text analysis algorithms to extract and understand the key content of the goals and tasks.

[1685] Step 6:

[1686] The server analyzes the media data (images, audio, video) it receives. Specifically, it applies image recognition, audio analysis, and video analysis algorithms to classify and understand the content of the media data.

[1687] Step 7:

[1688] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.). For example, it can recognize nervousness from the user's facial expressions during a presentation.

[1689] Step 8:

[1690] The server integrates the results of the text data analysis, media data analysis, and emotion engine analysis to generate a specific coaching plan for the user to achieve their goals and solve their problems. Specifically, it creates feedback, advice, and a specific action plan in the form of text and media based on the analysis results.

[1691] Step 9:

[1692] The server sends the generated feedback and advice to the device. Specifically, the feedback data is sent as an HTTP response, and the device receives it.

[1693] Step 10:

[1694] The feedback received by the device is displayed to the user. Specifically, a UI (user interface) is constructed to display the feedback content on the screen in an appropriate format, allowing the user to easily access it.

[1695] Step 11:

[1696] The server periodically tracks the user's progress. Specifically, each time the user uploads new data (e.g., a new presentation video), it receives and analyzes it and evaluates the user's progress.

[1697] Step 12:

[1698] The user periodically uploads new data (e.g., a new presentation video). Specifically, the user repeatedly sends progress data from the device to the server.

[1699] Step 13:

[1700] The server compares the results of the analysis with the previous results and generates updated feedback, specifically, updates the feedback based on the new data and provides it to the user.

[1701] Step 14:

[1702] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state. Specifically, it selects the optimal feedback method based on the user's profile information and applies an individual approach.

[1703] This allows the system to provide users with efficient and personalized coaching to help them improve their careers and performance.

[1704] Example 2

[1705] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1706] Effective feedback and coaching are essential for modern businesspeople to achieve their goals and challenges. However, traditional systems struggle to provide personalized feedback in real time that reflects users' emotions and progress. Furthermore, the lack of detailed progress tracking and updated feedback makes it difficult for users to continuously improve.

[1707] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for a user to input goals and tasks and upload text data and media data, means for receiving and saving the input data, means for analyzing the received data and generating feedback for the user to achieve their goals and solve their tasks, means for recognizing the user's emotions from their facial expressions, voice, and body movements and adjusting the feedback based on the results, and means for providing the generated feedback to the user. This makes it possible to provide personalized feedback that takes the user's emotional state into consideration, allowing the user to efficiently continue making improvements toward achieving their goals.

[1708] A "user" is an individual or organization that uses the system to achieve a goal or task.

[1709] A "goal" is a specific outcome or objective that a user wishes to achieve.

[1710] A "challenge" is a problem or obstacle that a user must overcome.

[1711] "Text data" refers to information or content expressed in a written format.

[1712] "Media data" refers to visual or audible information such as images, audio, and video, and includes non-text data.

[1713] "Upload" refers to sending data from the user's terminal to the server.

[1714] A "server" is a central system that receives, stores, and analyzes input data, and generates and provides feedback to users.

[1715] The "emotion engine" is a system that recognizes a user's emotional state by analyzing their facial expressions, voice, and body movements.

[1716] "Feedback" refers to advice and evaluations to help users achieve their goals and solve their problems.

[1717] "Tracking" is the act of continuously monitoring and recording a user's progress and analyzing that data.

[1718] "Personalization" means tailoring feedback content and communication style based on the individual personality and preferences of each user.

[1719] MODE FOR CARRYING OUT THE INVENTION

[1720] System Program Overview

[1721] This invention is a system that provides effective feedback and coaching to businesspeople to help them achieve their goals and challenges. Furthermore, by combining it with an emotion engine, the system adjusts the feedback based on the user's emotions. The following specific hardware and software are used to implement this system:

[1722] Hardware and software used

[1723] 1. On the user's device:

[1724] PC, smartphone, etc. Users can input their goals and tasks and upload text and media data.

[1725] 2. Server:

[1726] It acts as a central system, receiving, storing, and analyzing input data and providing feedback to users.

[1727] 3. Emotion Engine:

[1728] A system that recognizes emotions from a user's facial expressions, tone of voice, body movements, etc. For example, it uses facial expression recognition technology and voice analysis technology.

[1729] Program processing overview

[1730] Entering user information and initial settings

[1731] A user uses a terminal to input basic information about themselves (such as name, occupation, goals, and tasks). This input includes not only text data, but also image, audio, and video data as needed. For example, a user who aims to succeed in a new project may set improving their presentation skills as a task and upload videos of past presentations to the system.

[1732] Sending and Receiving Data

[1733] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[1734] Data analysis

[1735] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies, including natural language processing (NLP) of text data, as well as image recognition, speech analysis, and video analysis. For example, a presentation video can be analyzed to generate feedback on pronunciation and the flow of the presentation.

[1736] Emotion analysis

[1737] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.). For example, it can recognize nervousness from the user's facial expressions during a presentation.

[1738] Generate a coaching plan

[1739] The server generates a specific coaching plan for the user based on the analysis results and the emotion engine results. This plan includes feedback, advice, and a specific action plan. For example, a training plan for improving presentation skills could be created, with specific points to improve and practice methods.

[1740] Providing Feedback

[1741] The server generates feedback and advice, which is then sent to the device and provided to the user. Feedback is provided in the form of text, audio, video, etc., and the user can review and use it as appropriate. The user can also receive personalized feedback based on their emotional state.

[1742] Tracking progress

[1743] The server periodically tracks the user's progress: the user periodically uploads new data (e.g. a new presentation video), which the server analyzes and provides up-to-date feedback.

[1744] Personalized and adaptive

[1745] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state, resulting in more effective coaching. For example, if the user is nervous, the server can provide advice on how to relax.

[1746] Specific examples and examples of input prompts to the generative AI model

[1747] Specific examples

[1748] 1. A user aims to improve their presentation skills in order to succeed in a new project.

[1749] 2. The user uploads past presentation videos to the system from their device.

[1750] 3. The server receives the video and begins analyzing it, generating feedback using NLP, image recognition, and voice analysis technologies.

[1751] 4. The emotion engine recognizes the user's state of tension from the video and sends the emotional state to the server.

[1752] 5. The server generates feedback and gives specific advice, such as "Practice speaking slowly to improve your pronunciation. Take deep breaths when you feel nervous."

[1753] 6. When the user uploads a new video, feedback will be provided again.

[1754] Example of input prompt for generative AI model

[1755] Analyze the following data:

[1756] Name: Yamada Taro

[1757] Job type: Sales

[1758] Goal: New project success

[1759] Challenge: Improve presentation skills

[1760] Uploaded data: Past presentation videos

[1761] Use this data to generate feedback and a specific coaching plan, including advice on what to do if you're nervous.

[1762] This system allows users to receive highly efficient, personalized coaching at low cost, effectively improving their careers and performance.

[1763] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1764] Step 1: Entering user information and initial settings

[1765] A user logs into the system using a terminal and enters their basic information (name, job title, goals, challenges, etc.). This input includes not only text data, but also image, audio, video, and other data as needed. For example, a user whose goal is to succeed in a new project may set improving their presentation skills as a challenge and upload videos of past presentations to the system.

[1766] Input: User basic information, text data, media data

[1767] Output: Save input data

[1768] Specific behavior:

[1769] A user accesses the system's login page and logs in by entering authentication information (user name and password).

[1770] After logging in, a basic information input form is displayed and the user enters the required data.

[1771] Enter your goals and objectives, and upload media data (such as presentation videos).

[1772] Step 2: Sending and Receiving Data

[1773] The device sends the entered text data and uploaded media data to the server, which receives the data and stores it in a database.

[1774] Input: Data entered by the user

[1775] Output: Save data to database

[1776] Specific behavior:

[1777] When the user clicks the "Send" button, the terminal sends the data to the server.

[1778] The server processes the received data in real time and stores it in a database.

[1779] Step 3: Data analysis

[1780] The server analyzes the received data and generates feedback to help users achieve their goals and solve their problems. The analysis uses multimodal AI technologies such as image recognition, voice analysis, and video analysis, as well as natural language processing (NLP) of text data.

[1781] Input: User text data, media data

[1782] Output: Feedback generation

[1783] Specific behavior:

[1784] The server launches an NLP engine to analyze the meaning of the text data.

[1785] The video analysis engine analyzes the uploaded presentation video frame by frame and evaluates the user's speaking speed and clarity.

[1786] The voice analysis engine analyzes the tone and pace of the user's voice.

[1787] Step 4: Sentiment Analysis

[1788] The emotion engine recognizes emotions from the user's facial expressions, tone of voice, body movements, etc. Specifically, it analyzes video and audio data to identify emotional states (joy, anger, sadness, etc.).

[1789] Input: User's video data, audio data

[1790] Output: Emotion recognition result

[1791] Specific behavior:

[1792] The emotion engine uses facial expression recognition technology to analyze the user's facial features in the video.

[1793] The server receives the emotion analysis results and uses them to generate feedback.

[1794] Step 5: Create a coaching plan

[1795] The server generates a specific coaching plan for the user based on the results of the data analysis and the sentiment analysis, which includes feedback, advice, and a specific action plan.

[1796] Input: Data analysis results, emotion recognition results

[1797] Output: Coaching plan generated

[1798] Specific behavior:

[1799] The server generates detailed feedback on how to improve and practice your presentation.

[1800] The server will also provide relaxation techniques and advice to help relieve tension.

[1801] Step 6: Provide feedback

[1802] The server generates feedback and advice, which is sent to the device and provided to the user in the form of text, audio, video, etc.

[1803] Input: Generated feedback

[1804] Output: Providing feedback

[1805] Specific behavior:

[1806] The server generates feedback and sends it to the user's device as a push notification.

[1807] The feedback content is displayed on the terminal screen and the user can check it.

[1808] Step 7: Track your progress

[1809] The server periodically tracks the user's progress: the user periodically uploads new data (e.g. a new presentation video), which the server re-analyzes and provides updated feedback.

[1810] Input: User's new data

[1811] Output: Updated feedback

[1812] Specific behavior:

[1813] The server compares the user's past feedback with their current progress and generates an updated report.

[1814] When a user uploads a new presentation video, the server compares it with previous analysis results and identifies areas for improvement.

[1815] Step 8: Personalize and adapt

[1816] The server personalizes the feedback and communication style based on the user's personality, preferences, and emotional state, resulting in more effective coaching.

[1817] Input: User's personality profile, emotion recognition results

[1818] Output: Personalized feedback

[1819] Specific behavior:

[1820] The server adjusts the feedback content based on the user's personality profile and past feedback results.

[1821] Providing feedback based on emotional state (e.g., suggesting ways to relax if you're feeling tense).

[1822] (Application example 2)

[1823] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1824] There is a need for systems that provide feedback and coaching to improve the work efficiency of robotic workers in factories, reduce errors, and maintain optimal work conditions. However, conventional systems lack sufficient analysis of motion and sensor data, and are unable to provide appropriate feedback based on emotional states. This results in poor robot performance and has a negative impact on work efficiency and quality.

[1825] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1826] In this invention, the server includes means for a user to input goals and tasks and upload text data and media data, means for receiving and saving the input data, means for analyzing the received data and generating feedback for the user to achieve their goals and solve their tasks, means for providing the generated feedback to the user, means for analyzing video of the robot's work and sensor data and generating feedback for improving work efficiency and reducing errors, and means for performing emotional analysis based on the robot's sensor data and providing appropriate feedback according to the robot's emotional state. This makes it possible to analyze the robot's operating status in real time and provide optimal feedback and coaching.

[1827] A "user" is someone who uses the system or provides data to help the robotic worker achieve its goals and solve its problems.

[1828] "Goal" refers to the result or purpose that a user aims to achieve.

[1829] A "challenge" refers to a problem or obstacle that a user must overcome in the process of achieving a goal.

[1830] "Text data" refers to data that includes character information entered by the user.

[1831] "Media data" refers to data in various formats such as images, audio, and video.

[1832] "Reception" refers to the process in which the server acquires data sent from the user terminal.

[1833] "Storage" refers to recording the received data in a database or storage within the server.

[1834] "Analysis" is the process of extracting and evaluating information based on the received data to help the user achieve their goals and solve their problems.

[1835] "Feedback" refers to information or advice returned to the user based on the analysis results.

[1836] "Robot work video" is video data that records the movements of robots operating in a factory.

[1837] "Sensor data" refers to data such as temperature, vibration, and position obtained from sensors installed on the robot.

[1838] "Emotion analysis" is the process of inferring a robot's emotional state based on its movements and sensor data.

[1839] "Appropriate feedback" refers to the most effective advice and information based on the robot's state, taking into account the results of emotion analysis.

[1840] This invention is a system for improving the work efficiency and reducing errors of robot workers in factories. The system includes means for a user to input goals and tasks and upload text data and media data, means for receiving and saving the data, means for analyzing the received data, means for providing generated feedback to the user, means for analyzing video of the robot's work and sensor data to generate feedback for improving work efficiency and reducing errors, and means for performing emotional analysis and providing feedback according to the emotional state.

[1841] The server receives data input from the user's device (for example, information about the goals and challenges of robot workers in a factory, along with the associated text and media data) and stores it in a database. Software libraries such as TensorFlow and OpenCV are used to analyze the received data. These libraries are used to analyze motion in videos and sensor data, and generate the feedback necessary to help the user achieve their goals and solve their problems.

[1842] Furthermore, the server uses an emotion analysis engine to estimate the robot's emotional state (e.g., stress or fatigue) using video and sensor data from the robot's work. The emotion analysis engine evaluates the robot's movements and sensor data based on a specific emotion model and adjusts the feedback content. The generated feedback may include specific improvements, training modules, or maintenance schedules to improve work efficiency.

[1843] As a concrete example of this system, imagine a robot working on a conveyor line. Videos of the robot's work are recorded and sensor data (such as vibration and temperature data) is acquired. The server analyzes this data and provides feedback suggesting techniques to optimize the robot's movements, or, if high stress is detected from the sensor data, feedback suggesting when maintenance should be performed.

[1844] An example prompt might look like this:

[1845] "What operational improvements are needed to ensure that this conveyor line robot operates efficiently? Also, what maintenance is appropriate if it is experiencing fatigue or stress?"

[1846] This allows robotic workers in factories to receive optimal feedback and coaching in real time, improving work efficiency and reducing errors.

[1847] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1848] Step 1:

[1849] The server provides a means for users to input goals and tasks and upload text and media data. This allows users to set specific goals and tasks for the system and send data related to them. Input data could be, for example, video of robots working in a factory or sensor data. This input data is sent to the server and stored.

[1850] Step 2:

[1851] The server has a means for receiving and storing the input data. The received data includes text data (goals and tasks), video data (videos of the robot working), and sensor data (temperature, vibration, etc.). The received data is recorded in a database for subsequent analysis.

[1852] Step 3:

[1853] The server analyzes the received data. Specifically, it uses TensorFlow and OpenCV to analyze the movements in the video and extract important movement information from each frame. It also analyzes sensor data to determine the robot's operating status and detect abnormalities. This allows it to identify, for example, areas where the robot's movements are lagging or where abnormal vibrations are occurring.

[1854] Step 4:

[1855] The server provides the generated feedback to the user. Based on the analysis results, it generates specific advice on operation optimization techniques and areas for improvement. For example, if a specific operation is slow, it will suggest techniques to optimize that operation, and if an abnormality is detected, it will suggest maintenance.

[1856] Step 5:

[1857] The server performs emotion analysis based on the robot's work video and sensor data. The emotion analysis engine evaluates the video and sensor data to estimate the robot's emotional state (such as stress or fatigue). This allows it to provide individual feedback based on its condition.

[1858] Step 6:

[1859] The server then provides final feedback to the user. Taking into account the results of sentiment analysis, the server provides more personalized feedback to the user. For example, if high stress is detected, the server provides feedback recommending regular rest and maintenance.

[1860] Step 7:

[1861] Users receive feedback provided by the server and use it to improve the robot's work and for maintenance. This improves the robot's work efficiency and reduces errors. For example, by implementing the proposed motion optimization technology, the speed and accuracy of work can be improved. In addition, by performing regular maintenance, the robot's uptime can be maximized.

[1862] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1863] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1864] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1865] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1866] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1867] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1868] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1869] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1870] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1871] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1872] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1873] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1874] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1875] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1876] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1877] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1878] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1879] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1880] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1881] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1882] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1883] The following is further disclosed regarding the above embodiment.

[1884] (Claim 1)

[1885] means for users to input goals and challenges and upload text and media data;

[1886] means for receiving and storing the input data;

[1887] means for analyzing the received data and generating feedback for the user to achieve goals and solve problems;

[1888] means for providing said generated feedback to a user;

[1889] A system including:

[1890] (Claim 2)

[1891] 10. The system of claim 1, wherein said feedback means further comprises means for continuously tracking the user's progress and providing updated feedback based on said tracking results.

[1892] (Claim 3)

[1893] 10. The system of claim 1, further comprising means for personalizing said feedback content and communication style based on a user's personality and preferences.

[1894] (Claim 4)

[1895] 10. The system of claim 1, further comprising means for using multimodal AI to analyze not only text data but also image, audio, and video data.

[1896] (Claim 5)

[1897] 10. The system of claim 1, further comprising means for providing said feedback and advice to the user in real time or on a pre-set schedule.

[1898] "Example 1"

[1899] (Claim 1)

[1900] means for users to input goals and challenges and upload text and media data;

[1901] means for receiving and storing the input data;

[1902] means for analyzing the received data and generating feedback for the user to achieve goals and solve problems using natural language processing, image recognition, voice analysis, and video analysis;

[1903] means for providing said generated feedback to a user;

[1904] A system including:

[1905] (Claim 2)

[1906] 10. The system of claim 1, wherein said feedback means further comprises means for continuously tracking the user's progress and providing updated feedback based on said tracking results.

[1907] (Claim 3)

[1908] 10. The system of claim 1, further comprising means for personalizing said feedback content and communication style based on a user's personality and preferences.

[1909] "Application Example 1"

[1910] (Claim 1)

[1911] means for users to input goals and challenges and upload text and media data;

[1912] means for receiving and storing the input data;

[1913] means for analyzing the received data and generating feedback for the user to achieve goals and solve problems;

[1914] means for providing said generated feedback to a user;

[1915] A means of recording factory workers' movements with cameras and providing real-time feedback;

[1916] A system including:

[1917] (Claim 2)

[1918] 10. The system of claim 1, wherein the feedback means further comprises means for continuously tracking the user's progress and providing updated feedback based on the tracking results.

[1919] (Claim 3)

[1920] 10. The system of claim 1, further comprising means for personalizing said feedback content and communication style based on a user's personality and preferences.

[1921] "Example 2: Combining Emotion Engines"

[1922] (Claim 1)

[1923] means for users to input goals and challenges and upload text and media data;

[1924] means for receiving and storing the input data;

[1925] means for analyzing the received data and generating feedback for the user to achieve goals and solve problems;

[1926] A means for recognizing emotions from the user's facial expressions, voice, and body movements, and adjusting feedback based on the results;

[1927] means for providing said generated feedback to a user;

[1928] A system including:

[1929] (Claim 2)

[1930] 10. The system of claim 1, wherein said feedback means further comprises means for continuously tracking the user's progress and providing updated feedback based on said tracking results.

[1931] (Claim 3)

[1932] 10. The system of claim 1, further comprising means for personalizing said feedback content and communication style based on a user's personality and preferences.

[1933] "Application example 2 when combining emotion engines"

[1934] (Claim 1)

[1935] means for users to input goals and challenges and upload text and media data;

[1936] means for receiving and storing the input data;

[1937] means for analyzing the received data and generating feedback for the user to achieve goals and solve problems;

[1938] means for providing said generated feedback to a user;

[1939] A method for analyzing robot work videos and sensor data to generate feedback for improving work efficiency and reducing errors.

[1940] A means for performing emotion analysis based on the sensor data of the robot and providing appropriate feedback according to the robot's emotional state;

[1941] A system including:

[1942] (Claim 2)

[1943] 10. The system of claim 1, wherein said feedback means further comprises means for continuously tracking the user's progress and providing updated feedback based on said tracking results.

[1944] (Claim 3)

[1945] 10. The system of claim 1, further comprising means for personalizing said feedback content and communication style based on a user's personality and preferences. [Explanation of symbols]

[1946] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for users to input goals and challenges and upload text and media data; means for receiving and storing the input data; means for analyzing the received data and generating feedback for the user to achieve goals and solve problems; means for providing said generated feedback to a user; A system including:

2. 2. The system of claim 1, wherein said feedback means further comprises means for continuously tracking the user's progress and providing updated feedback based on said tracking results.

3. 10. The system of claim 1, further comprising means for personalizing said feedback content and communication style based on a user's personality and preferences.

4. 10. The system of claim 1, further comprising means for using multimodal AI to analyze not only text data but also image, audio, and video data.

5. The system of claim 1 further comprising means for providing said feedback and advice to the user in real time or on a pre-set schedule.

Images