system

Abstract

Provide a system. [Solution] A means of collecting employee work behavioral patterns and performance data in real time; a means for storing the collected data in a database; means for preprocessing the stored data and analyzing it using a performance evaluation model; means for generating a personalized feedback message based on the analysis results; a means for delivering the generated feedback to an employee's device; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's workplace, it is important to efficiently manage employee performance and skill development. However, traditional methods require managers to manually evaluate and provide feedback, which can be time-consuming and labor-intensive, leading to issues with subjectivity and fairness in evaluations. It can also be difficult for employees to clearly identify their specific areas for improvement and growth opportunities. Therefore, there is a need for a system that automates the evaluation and feedback process and supports employee skill development. [Means for solving the problem]

[0005] The present invention provides a system including: a means for collecting employee behavioral patterns and performance data during work in real time; a means for storing the collected data in a database; a means for preprocessing the stored data and analyzing it using a performance evaluation model; a means for generating individual feedback messages based on the analysis results; and a means for delivering the generated feedback to employee terminals. This system automates the evaluation and feedback process, reducing the burden on managers and enabling the provision of personalized, specific improvement points and growth opportunities to employees.

[0006] An "employee" refers to a person who belongs to a company or organization and performs designated tasks or duties.

[0007] "Work behavior patterns" refers to the series of actions and behaviors that employees perform when performing their work, such as the start time of a task, the time it is completed, how it is performed, and break tendencies.

[0008] "Performance Data" means quantitative and qualitative information about an employee's job performance, including time to complete a task, number of errors, and work efficiency.

[0009] "Real-time" refers to the state in which data and information are collected, processed, and transmitted almost simultaneously, and are reflected immediately without delay.

[0010] "Collection Instrument" refers to a device, software, or method for collecting particular data.

[0011] A "database" refers to a structured information system for efficiently storing, retrieving, and managing large amounts of data.

[0012] "Preprocessing" refers to the cleaning, transformation, screening, and other processes performed on raw data to convert it into an analyzable format.

[0013] "Performance Evaluation Model" means a collection of statistical or machine learning algorithms used to analyze and evaluate employee performance.

[0014] "Analytical tools" refer to techniques and methods used to derive meaningful information and results from collected data.

[0015] A "feedback message" refers to a message that specifically provides information based on the analysis results, including an evaluation of an employee's performance and suggestions for improvement.

[0016] "Delivery means" refers to the equipment, software, or method for transmitting or notifying generated information or messages to a specific recipient (in this case, an employee's device).

[0017] "Terminal" refers to a device such as a computer, smartphone, or tablet that is directly operated by a user. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] The present invention provides a system for evaluating employee performance and providing personalized feedback. A specific embodiment of this system will be described below.

[0040] 1. Data Collection

[0041] The device collects data in real time while the user is working. Specifically, it records behavioral patterns and performance data such as task completion time, number of errors, and operation methods used. For example, the moment a user completes a specific task, the device collects detailed data such as "task completion time: 20 minutes" and "number of errors: 2."

[0042] 2. Data storage

[0043] The device sends the collected data at regular intervals to a server, which stores the received data in a database that stores the behavioral patterns and performance data of many users.

[0044] 3. Data Preprocessing

[0045] The server preprocesses the raw data stored in the database, which includes cleaning the data (e.g., imputing missing values ​​and removing outliers), normalizing it, and extracting necessary features. The preprocessed data is then converted into a format suitable for analysis.

[0046] 4. Performance Analysis

[0047] The server analyzes the pre-processed data using a machine learning algorithm. Specifically, it uses a pre-trained performance evaluation model (e.g., a random forest classifier) ​​to predict user performance. This analysis results in a performance evaluation based on each user's behavioral patterns.

[0048] 5. Generate feedback

[0049] The server generates personalized feedback messages based on the analysis, including specific instructions such as "You're performing well, keep it up" or "You're making a lot of errors, so you need to pay attention to a particular skill."

[0050] 6. Feedback Distribution

[0051] The server sends the generated feedback to the terminal, which notifies the user and displays it on the screen, allowing the user to understand their own performance and take specific action to improve.

[0052] Specific examples

[0053] Consider a scenario where User A completes a task in 20 minutes with 2 errors.

[0054] The device collects this data and sends it to the server every 10 minutes.

[0055] The server stores the received data in a database.

[0056] After the data is saved, the server cleans it, removes outliers, and then analyzes it using a machine learning model.

[0057] The analysis results in an evaluation of "good performance."

[0058] The server generates feedback saying, "Performance is good, keep it up."

[0059] Finally, the terminal displays this feedback to User A, who then decides on the next action to take based on the feedback.

[0060] This system automates the evaluation and feedback process, significantly reducing the burden on managers and efficiently supporting employee skill improvement.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The user begins work and performs tasks on their device, such as creating documents and entering data for a new project.

[0064] Step 2:

[0065] The device records user behavior patterns and performance data in real time, including task start and end times, operation types, and the number of errors.

[0066] Step 3:

[0067] The device sends the collected data to the server at regular intervals, for example, uploading the data in batches every 10 minutes.

[0068] Step 4:

[0069] The server stores the received data in a database, allowing the behavioral patterns and performance data of all employees to be centrally managed.

[0070] Step 5:

[0071] The server periodically preprocesses the data stored in the database, which includes data cleaning (e.g., imputing missing values, removing outliers), normalization, feature extraction, etc.

[0072] Step 6:

[0073] The server uses the preprocessed data to analyze it with a machine learning algorithm (e.g., random forest classifier), which then evaluates performance based on user behavior patterns.

[0074] Step 7:

[0075] The server generates personalized feedback messages based on the analysis, such as "Your performance is good, keep it up" or "Your number of errors is high, so you need to pay attention to a particular skill."

[0076] Step 8:

[0077] The server transmits the generated feedback message to the user's terminal.

[0078] Step 9:

[0079] The device notifies the user of the received feedback and displays it on the screen, allowing the user to view the feedback and check their own performance.

[0080] Step 10:

[0081] Users take specific action to improve based on the feedback, such as working more carefully to reduce the number of errors or training in a particular skill.

[0082] This series of steps automates the evaluation and feedback process, providing users with concrete guidelines for improvement.

[0083] Example 1

[0084] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0085] Objectively and efficiently evaluating employee performance and providing personalized feedback is important for helping employees improve their skills and increasing work efficiency. However, traditional evaluation methods often involve subjective judgments and manual evaluations, which require time and effort, making it difficult to provide fast and accurate feedback. Another issue is that it is difficult to generate specific feedback for each employee that identifies areas for improvement.

[0086] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0087] In this invention, the server includes means for collecting user behavioral patterns and performance data during work in real time, means for transmitting the collected data at regular intervals, means for storing the transmitted data in a database, means for preprocessing the stored data, means for analyzing the preprocessed data using a performance evaluation model, means for generating individual feedback messages based on the analysis results, and means for delivering the generated feedback to the user's terminal, thereby enabling automatic and efficient evaluation of employee performance and rapid provision of individual, specific feedback.

[0088] "User" means an individual who uses the System and provides behavioral pattern and performance data.

[0089] A "behavioral pattern" refers to a series of actions or movements that a user makes when operating a system, and includes, for example, clicking operations and key inputs.

[0090] "Performance data" refers to specific numerical data for evaluating a user's work efficiency and work results, such as the time it takes a user to complete a task or the number of errors.

[0091] "Means of collecting data in real time" refers to methods of immediately acquiring and recording data when a user performs an operation, including the use of sensors or software logging functions.

[0092] "Means for sending at regular intervals" refers to means for sending collected data to a server at regular time intervals (e.g., every 10 minutes).

[0093] "Means of storing data in a database" refers to a system or method for systematically organizing and safely storing collected data, such as using a relational database management system (RDBMS).

[0094] "Preprocessing means" refers to a method for removing noise from stored raw data and converting it into a format suitable for analysis, including, for example, imputing missing values ​​and removing outliers.

[0095] "Performance Evaluation Model" means a machine learning algorithm model used to evaluate a user's work efficiency and work results based on collected and pre-processed data, such as a random forest classifier.

[0096] "Means for analyzing" refers to the method of inputting the pre-processed data into a performance evaluation model to analyze and predict user performance.

[0097] A "feedback message" refers to a specific message that is generated based on the analysis results and includes an evaluation of the user and points for improvement.

[0098] "Means for delivering to terminal" refers to a method for transmitting the generated feedback message to a terminal accessible to the user.

[0099] The present invention provides a system for evaluating employee performance and providing personalized feedback. A specific embodiment of this system will be described below.

[0100] This system is mainly composed of three entities: a server, a terminal, and a user.

[0101] Data collection

[0102] The device collects data in real time while the user is working. Specifically, it captures detailed performance data such as the actions the user takes (clicks, typing, movement, etc.), the time it takes to complete a task, and the number of errors. This collection is done using software logging functions and sensors. For example, if a user completes a specific task over a 20-minute period and two errors occur during that time, the device will record this data.

[0103] Sending data

[0104] The device sends the collected data to the server at regular intervals (for example, every 10 minutes). The data is sent using the HTTP protocol and encoded in JSON format.

[0105] Data storage

[0106] The server stores the received data in a database. This database stores the behavioral patterns and performance data of each user. For the database, a relational database such as MySQL (registered trademark) or PostgreSQL is used.

[0107] Data Preprocessing

[0108] The server preprocesses the raw data stored in the database. Specific preprocessing includes filling in missing values, removing outliers, and normalizing the data. It also extracts the features necessary for analysis. For example, for raw data retrieved from the database, it fills in missing values ​​with the average value, filters outliers, and normalizes each data item to a standard scale.

[0109] Performance Analysis

[0110] The server then uses the pre-processed data to analyze it with a machine learning algorithm, which uses a pre-trained random forest classifier to predict user performance. The pre-processed data is input into the random forest model to calculate a performance evaluation score for each user.

[0111] Generate feedback

[0112] The server generates a personalized feedback message based on the analysis results, including specific instructions such as "Your performance is good. Keep it up," or "Your number of errors is high, so you need to pay attention to a specific skill." This feedback message is dynamically generated using a feedback template based on the analysis results using an embedded sentence generation model.

[0113] Feedback Distribution

[0114] The server sends the generated feedback to the device, which notifies the user and displays it on the screen. The server also sends the generated feedback message in JSON format to the device, which parses the message and displays it in the notification area or on the dashboard.

[0115] Specific examples

[0116] Here is a specific example where User A completed a task in 20 minutes with 2 errors.

[0117] 1. The device records user A's actions and sends the data to the server every 10 minutes.

[0118] 2. The server stores the received data in a database.

[0119] 3. After the data is saved, the server cleans the data and removes outliers, then inputs the preprocessed data into a random forest model for analysis.

[0120] 4. The analysis results show that the performance is good.

[0121] 5. The server generates a feedback message saying, "Performance is good, keep it up."

[0122] 6. Finally, the device displays this feedback to User A, who then decides on the next action.

[0123] This prompt explains the overall system flow and the technical details of each process, automating the employee evaluation and feedback process, reducing the burden on managers and efficiently helping employees improve their skills.

[0124] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0125] Step 1: Data collection

[0126] The device collects data in real time while the user is working. This includes detailed performance data such as actions performed by the user, such as clicking, typing, and moving, as well as the time it took to complete a task and the number of errors. Specifically, when a user clicks the "Save" button, the click event is recorded, and if an error occurs, an error message is also added to the log. This collected data is the input data.

[0127] Input: User action data (clicks, typing, movement), task completion time, number of errors

[0128] Output: Detailed performance data logged

[0129] Step 2: Sending data

[0130] The device sends the collected data to the server at regular intervals (for example, every 10 minutes). The data is encoded in JSON format using the HTTP protocol and sent. Specifically, a timer event occurs every 10 minutes, and the device sends the collected data to the server as an HTTP request.

[0131] Input: Detailed logged performance data

[0132] Output: JSON format data sent to the server

[0133] Step 3: Save your data

[0134] The server stores the received data in a database, which stores each user's behavioral patterns and performance data. Specifically, the server parses the received JSON data and inserts each field into the corresponding table in the database.

[0135] Input: JSON format data sent to the server

[0136] Output: Behavioral patterns and performance data stored in a database

[0137] Step 4: Preprocessing the data

[0138] The server preprocesses the raw data stored in the database. This preprocessing includes filling missing values, removing outliers, normalizing the data, and extracting necessary features. Specifically, for the raw data retrieved from the database, the server fills missing values ​​with the mean value, filters out outliers, and then normalizes each data to a standard scale.

[0139] Input: Behavioral patterns and performance data stored in a database

[0140] Output: Preprocessed data

[0141] Step 5: Analyze performance

[0142] The server then uses the preprocessed data to analyze it with a machine learning algorithm, using a pre-trained random forest classifier to evaluate user performance. Specifically, the preprocessed data is input into the random forest model to calculate a performance evaluation score for each user.

[0143] Input: Preprocessed data

[0144] Output: Performance evaluation score

[0145] Step 6: Generate feedback

[0146] The server generates individual feedback messages based on the analysis results. This evaluates the user's performance based on the analysis results, and dynamically generates feedback templates based on specific improvements and instructions using an embedded sentence generation model. For example, it generates a message such as "Your performance is good. Please keep up the good work."

[0147] Input: Performance evaluation score

[0148] Output: Individual feedback messages

[0149] Step 7: Deliver feedback

[0150] The server sends the generated feedback message to the user's device. The device notifies the user of this feedback and displays it on the screen. Specifically, the server sends the generated feedback message to the device in JSON format, and the device parses the message and displays it in the notification area or on the dashboard.

[0151] Input: Individual feedback message

[0152] Output: Feedback message displayed on the screen

[0153] (Application example 1)

[0154] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0155] Traditionally, employee work efficiency and performance evaluations have often been done manually, which not only lacks objectivity but also requires a great deal of time and effort. Feedback based on evaluation results is also common, making it difficult to provide personalized feedback to individual employees. This makes it difficult to improve employee skills and maintain motivation. This is particularly true in logistics centers and other workplaces, where real-time evaluations and immediate feedback are required, and traditional methods are insufficient.

[0156] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0157] In this invention, the server includes means for collecting employee behavioral patterns and performance data in real time during work, means for storing the collected data in a database, means for preprocessing the stored data and analyzing it using a performance evaluation model, means for generating individual feedback messages based on the analysis results, means for delivering the generated feedback to the employee's terminal, and means for transmitting data to smart glasses that display the feedback messages, thereby enabling real-time work evaluation of employees and immediate individual feedback.

[0158] "Employee" means a person employed to perform a specific job.

[0159] A "behavioral pattern" is a series of actions or movements that an employee tends to take while working.

[0160] "Performance data" refers to data that numerically indicates an employee's work efficiency and results.

[0161] "Real-time" refers to immediacy, with almost no time lag between data collection and processing.

[0162] A "database" is a digital storage device that allows collected data to be stored, managed, and searched in an organized manner.

[0163] "Preprocessing" refers to the process of cleaning data and preparing it into the required format for data analysis.

[0164] A "performance evaluation model" is a mathematical or algorithmic method for analyzing and evaluating employee performance.

[0165] "Analysis" refers to information processing that derives meaning and trends from collected data.

[0166] A "personalized feedback message" is a message that provides specific improvement suggestions or evaluations based on a specific employee's performance.

[0167] "Employee device" means a digital device used by an employee in the course of work.

[0168] "Distribution" refers to the act of electronically transmitting a generated feedback message.

[0169] "Smart glasses" are eyeglass-type digital devices that have a display function and can provide information visually.

[0170] This invention is a system that evaluates the work efficiency and performance of employees in real time at work sites such as logistics centers and provides instant personalized feedback. The system uses smart glasses to collect data while employees are working, analyzes the data on a server, and generates and provides feedback to the smart glasses.

[0171] First, employees wear smart glasses while performing their tasks. The glasses collect real-time data on the employee's behavioral patterns and performance (e.g., work time, number of errors, etc.). This data is collected through an interface within the smart glasses.

[0172] The smart glasses then send the collected data to a server at regular intervals, which stores it in a database that stores the behavioral patterns and performance data of numerous employees.

[0173] The server preprocesses the raw data stored in the database, including cleaning the data (e.g., imputing missing values ​​and removing outliers), normalizing it, and extracting necessary features. The preprocessed data is then converted into a format suitable for analysis.

[0174] The server then uses machine learning algorithms (e.g., random forest classifiers or generative AI models) to analyze the preprocessed data. This results in a performance evaluation based on each employee's behavioral patterns. The performance evaluation model is designed to provide specific evaluations and areas for improvement to improve employees' skills.

[0175] Based on the analysis results, an individual feedback message is generated. The feedback message includes performance evaluation results, areas for improvement, and specific advice for skill improvement. For example, feedback such as "Your picking efficiency is high and you are performing well. Please continue," or "There are many errors, so you need to be careful with certain operations."

[0176] The generated feedback is sent from the server to the smart glasses, which visually display the feedback and allow employees to instantly recognize it, allowing them to understand their own performance in real time and take specific action to improve.

[0177] Specific examples

[0178] Let's say that at a logistics center, Employee A is wearing smart glasses while performing picking work. The smart glasses record the time it takes Employee A to pick up a specific item from a shelf and the number of errors he makes. This data is sent to a server in real time and stored in a database.

[0179] The server preprocesses the data and evaluates Employee A's performance using random forests. The analysis results show that Employee A's picking efficiency is high and the number of errors is low, so feedback is generated saying, "Your picking efficiency is high and your performance is excellent. Please continue." This feedback is sent to the smart glasses so that Employee A can check it while he works.

[0180] Prompt Sentence Examples

[0181] "Data is collected when employees complete specific tasks and sent to a server. The data is cleaned and analyzed using machine learning models. Picking efficiency is evaluated and the results are displayed as feedback on the smart glasses."

[0182] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0183] Step 1:

[0184] The device (smart glasses) collects behavioral patterns and performance data in real time while employees are working. The input is employee movement data, and the output is that this data is temporarily stored in the device. Specific operations include recording employee movements and operations with sensors and collecting data such as the time it takes to complete a task and the number of errors.

[0185] Step 2:

[0186] The device sends the collected data to the server at regular intervals. The input is the behavioral patterns and performance data temporarily stored in the device, and the output is the transmission of this data to the server. Specifically, the data is uploaded from the device to the server via wireless communication.

[0187] Step 3:

[0188] The server stores the received data in a database. The input is the behavioral patterns and performance data received from the terminal, and the output is the accumulation of this data in the database. Specifically, the server periodically writes the data to the database.

[0189] Step 4:

[0190] The server preprocesses the stored data. The input is the raw data stored in the database, and the output is the preprocessed data. Specific operations include cleaning the data (filling in missing values ​​and removing outliers) and normalizing it.

[0191] Step 5:

[0192] The server analyzes the preprocessed data using a performance evaluation model. The input is the preprocessed data, and the output is the employee performance evaluation results. Specifically, the server analyzes the data using a machine learning algorithm (e.g., a random forest classifier or a generative AI model) to evaluate each employee's performance.

[0193] Step 6:

[0194] The server generates individual feedback messages based on the analysis results. The input is the performance evaluation results, and the output is the feedback message. Specifically, the server generates a personalized feedback message for each employee based on the analysis results.

[0195] Step 7:

[0196] The server delivers the generated feedback to the terminal. The input is the feedback message, and the output is the feedback message sent to the terminal. As a specific operation, the feedback message is sent to the smart glasses and displayed.

[0197] Step 8:

[0198] The terminal visually displays the received feedback. The input is the feedback message sent from the server, and the output is the feedback message displayed to the employee. Specifically, the feedback message is displayed as a pop-up on the display of the smart glasses.

[0199] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0200] This invention combines a system that collects and analyzes employee behavioral patterns and performance data in real time to provide individualized feedback with an emotion engine that recognizes the user's emotions. This system enables performance evaluation and feedback that take into account the employee's emotional state.

[0201] 1. Data Collection

[0202] The user starts work and performs tasks on their device, such as creating documents for a new project or entering data. The device records the employee's behavioral patterns and performance data (e.g., task completion time, number of errors, etc.) in real time. In addition, the device is equipped with a camera and microphone, which are used to collect the user's facial expressions and voice.

[0203] 2. Emotion recognition

[0204] The device sends the collected facial and voice data to the emotion engine, which uses machine learning algorithms to analyze the user's emotional state (e.g., joy, sadness, anger, surprise, etc.). The analysis results are also recorded.

[0205] 3. Data storage

[0206] The device periodically transmits collected data (behavioral patterns, performance data, and emotion data) to the server, which stores the data in a database.

[0207] 4. Data Preprocessing

[0208] The server preprocesses the raw data stored in the database, including data cleaning, normalization, and feature extraction, and also converts the sentiment data into a format suitable for analysis.

[0209] 5. Performance Analysis

[0210] The server then uses the preprocessed data to analyze it using a machine learning algorithm (e.g., a random forest classifier). This also takes into account the user's emotional data. For example, high levels of stress or frustration can affect performance evaluation.

[0211] 6. Generate feedback

[0212] The server generates a personalized feedback message based on the analysis results, and adjusts the tone and wording of the feedback based on the emotional data. For example, it might say, "You're performing well, but you seem a little stressed. Please take a break and keep working hard."

[0213] 7. Feedback Distribution

[0214] The server sends the generated feedback message to the device, which notifies the user and displays it on the screen. The user can view this feedback to understand both their own performance and emotional state.

[0215] Specific examples

[0216] Consider a scenario where User A completes a task in 20 minutes with 2 errors.

[0217] The device collects this data, and also uses a camera and microphone to capture the user's facial expressions and voice.

[0218] The emotion engine generates an analysis result that "the user is feeling a little stressed."

[0219] The terminal transmits this data to the server.

[0220] The server stores the data in a database and performs preprocessing.

[0221] The machine learning model predicts that "performance is good, but there is some stress."

[0222] The server generates feedback like, "You're performing well, but you seem a little stressed. Please take a break and keep trying."

[0223] The terminal notifies User A of this feedback and displays it on the screen.

[0224] User A takes the feedback into consideration, takes a break, and works on the next task.

[0225] This system automates the evaluation and feedback process, significantly reducing the burden on managers and efficiently supporting employees in improving their skills and managing their emotions.

[0226] The processing flow will be explained below.

[0227] Step 1:

[0228] The user begins work and performs tasks on their device, such as creating documents and entering data for a new project.

[0229] Step 2:

[0230] The device collects user behavioral patterns and performance data in real time. Specifically, it records the task start time, completion time, operation type, number of errors, etc. For example, if a user completes a task in 20 minutes and two errors occur, the device will record that data.

[0231] Step 3:

[0232] The device also uses its built-in camera and microphone to collect facial and audio data, such as recording changes in facial expression and tone of voice while the user performs a task.

[0233] Step 4:

[0234] The terminal periodically (for example, every 10 minutes) transmits the collected behavioral patterns, performance data, and emotion data to the server.

[0235] Step 5:

[0236] The server stores the received data in a database, including, for example, the user's task completion time, number of errors, facial expression data, and tone of voice.

[0237] Step 6:

[0238] The server preprocesses the stored data, which includes cleaning the data (filling in missing values ​​and removing outliers), normalizing the data, and extracting necessary features, such as emotional features (tension, stress, etc.) from speech data.

[0239] Step 7:

[0240] The server uses the preprocessed data to analyze it with a machine learning algorithm (e.g., a random forest classifier) ​​to predict the user's performance. This analysis takes into account not only behavioral patterns and performance data, but also emotional data.

[0241] Step 8:

[0242] The server generates individual feedback messages based on the analysis results. For example, if the user's performance is good, the server generates a message saying, "Your performance is good. Keep up the good work." In addition, based on the emotional data, the server may generate a message saying, "You seem to be feeling a bit stressed. Take a break and keep working hard."

[0243] Step 9:

[0244] The server sends the generated feedback message to the terminal, for example, the above feedback message to the user's terminal.

[0245] Step 10:

[0246] The device notifies the user of the received feedback message and displays it on the screen, for example, as a pop-up notification or on the dashboard.

[0247] Step 11:

[0248] The user reviews the feedback message and takes specific actions to improve their behavior, such as working more carefully to reduce the number of errors or taking breaks.

[0249] This series of steps automates the evaluation and feedback process, reducing the burden on managers and helping them manage both their employees' performance and their emotional state.

[0250] Example 2

[0251] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0252] While traditional employee evaluation systems collect and evaluate behavioral patterns and performance data, they do not take into account the emotional state of employees, making it difficult to provide appropriate feedback based on the psychological state of employees. Furthermore, by taking emotional factors into account in performance evaluations, it is expected that employee motivation and stress management will be improved, optimizing overall performance.

[0253] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0254] In this invention, the server includes means for collecting employee behavioral patterns and performance data during work in real time, means for transmitting the collected facial and voice data to an emotion recognition engine and analyzing the emotional state, means for storing the collected data and analysis results in a database, means for preprocessing the stored data and analyzing it using a performance evaluation model, means for generating individual feedback messages based on the analysis results, and means for delivering the generated feedback to the employee's terminal. This makes it possible to combine the employee's emotional state with the performance evaluation and provide more accurate and individual feedback.

[0255] "Behavioral patterns" refer to a series of actions and behavioral tendencies exhibited by employees during work hours, and include operation logs and time management data.

[0256] "Performance data" refers to indicators of the results and efficiency of an employee's work, including specific data such as the time it takes to complete a task or the number of errors made.

[0257] "Emotion recognition engine" refers to machine learning models and algorithms that analyze collected facial and voice data to identify an employee's emotional state (e.g., joy, sadness, anger, etc.).

[0258] "Emotional state" refers to the psychological state that employees feel while working, and includes joy, sadness, anger, surprise, stress, etc.

[0259] "Database" refers to a system for efficiently storing and managing collected behavioral patterns, performance data, and emotional state data.

[0260] "Preprocessing" refers to a series of operations performed to convert stored raw data into a form suitable for analysis, including, for example, data cleaning, normalization, and feature extraction.

[0261] A "performance evaluation model" refers to a machine learning algorithm or analytical method used to evaluate an employee's job performance and results.

[0262] "Feedback messages" refer to messages containing advice and evaluations provided to employees based on the analysis results, and indicate specific areas for improvement and opportunities for growth.

[0263] "Devices" refers to devices such as computers and smartphones used by employees to carry out their work, and are used to collect data and display feedback.

[0264] MODE FOR CARRYING OUT THE INVENTION

[0265] This invention combines a system that collects and analyzes employee behavioral patterns and performance data in real time to provide individualized feedback with an emotion engine that recognizes the user's emotions. This system enables performance evaluation and feedback that take into account the employee's emotional state.

[0266] 1. Data Collection

[0267] The user begins work and performs a task on the device, such as creating documents for a new project or entering data. The device records the user's operation log (clicks, keystrokes, etc.) in real time and collects performance data such as the time it took to complete the task and the number of errors. The device is equipped with a built-in camera and microphone, which are used to collect the user's facial expressions and voice. During this process, dedicated software installed on the device runs and saves the data in an appropriate format.

[0268] 2. Emotion recognition

[0269] The device sends the collected facial and voice data to an emotion recognition engine. This emotion engine uses machine learning models to analyze the user's emotional state in real time. Specifically, it uses deep learning to analyze the collected facial data and classify emotions such as "happiness," "sadness," and "anger." Similarly, it analyzes the voice data using voice recognition technology to analyze emotions. The analysis results of the emotion recognition engine are returned to the device as a dataset containing the user's emotional state, which is also recorded.

[0270] 3. Data storage

[0271] The device periodically sends collected data (behavioral patterns, performance data, and emotional data) to the server. The data is sent in encrypted form to ensure security. The server stores the received data in a database. The SQL queries and data format used at this time are in accordance with the system specifications.

[0272] 4. Data Preprocessing

[0273] The server cleans the raw data stored in the database, for example by imputing missing values ​​and detecting and correcting outliers. It also performs data normalization (for example, scaling and encoding categorical data). Specific preprocessing steps are also performed on the emotion data to convert it into a format suitable for analysis (for example, feature extraction and time series data shaping).

[0274] 5. Performance Analysis

[0275] The server then uses the preprocessed data to analyze it with a machine learning algorithm. Specifically, it uses a random forest classifier to evaluate employee performance. Emotional data is also used as a feature to influence the model's predictions. For example, if a user is under high stress, their task efficiency tends to decrease, and this is also reflected in the model.

[0276] 6. Generate feedback

[0277] The server generates personalized feedback messages based on the analysis results, using natural language generation (NLG) technology to select appropriate tones and expressions based on emotional data. For example, a message might be generated that reads, "Your performance is good, but you seem to be feeling a bit stressed. Please take breaks and keep working hard."

[0278] 7. Feedback Distribution

[0279] The server sends the generated feedback message to the device. The device receives it and notifies the user. The device displays the feedback on the screen and notifies the user of the recorded content by voice notification, etc. When the user sees the feedback, they can adjust their behavior based on the content.

[0280] Specific examples

[0281] Consider a scenario in which User A begins a task and completes it in 20 minutes, with two errors. The device collects this data in real time, simultaneously capturing the user's facial expressions and voice using a camera and microphone. The device sends the facial and voice data to an emotion recognition engine, which generates an analysis result indicating that "User A is feeling a little stressed." The device receives this result and sends it to the server. The server stores it in a database and preprocesses the data using Python's Pandas library. Outlier detection algorithms and data cleaning techniques are used. A random forest classifier predicts that User A's performance is "good, but appears to be slightly stressed." NLG technology is used to generate a feedback message stating, "Your performance is good, but you appear to be feeling a little stressed. Please take a break and continue working hard." The server then sends the message to the device, which displays it on the device's screen, informing User A. User A can take this information into consideration and take appropriate measures, such as taking a break.

[0282] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0283] Step 1:

[0284] The user begins work and performs tasks on the device. The user creates documents for a new project, enters data, etc. The device records the user's operation log (clicks, keystrokes, etc.) in real time and collects performance data such as task completion time and number of errors. This data is entered and temporarily stored on the device.

[0285] Step 2:

[0286] The device uses a built-in camera and microphone to collect the user's facial and voice data. This data is also recorded in real time and saved in an appropriate format by dedicated software. This collected facial and voice data is input and temporarily stored within the device.

[0287] Step 3:

[0288] The device sends the collected facial and voice data to an emotion recognition engine, which uses machine learning models to analyze the user's emotional state (happiness, sadness, anger, etc.) in real time. The input for the analysis is facial and voice data, and the output is data indicating the user's emotional state. The analysis results are returned to the device and recorded.

[0289] Step 4:

[0290] The device periodically transmits all collected data (behavioral patterns, performance data, and emotion data) to the server. This uses an encrypted protocol to ensure data security. The transmitted data is the input, and the data received by the server is the output.

[0291] Step 5:

[0292] The server stores the received data in a database, recording the data in an SQL query or appropriate format. The input is the data sent and the output is the data stored in the database.

[0293] Step 6:

[0294] The server preprocesses the raw data stored in the database. It performs missing value imputation, outlier detection and correction, and data normalization (scaling and encoding). The input is raw data, and the output is preprocessed data.

[0295] Step 7:

[0296] The server uses the preprocessed data to perform analysis using a machine learning algorithm and a random forest classifier to evaluate employee performance. The inputs are the preprocessed data and sentiment data, and the output is the performance evaluation results.

[0297] Step 8:

[0298] The server generates individual feedback messages based on the analysis results. It uses natural language generation (NLG) technology to select appropriate tones and expressions based on the emotional data. The input is the performance evaluation results and emotional data, and the output is the feedback message.

[0299] Step 9:

[0300] The server sends the generated feedback message to the terminal, which receives it and notifies the user. The input is the feedback message, and the output is the notification to the user.

[0301] Step 10:

[0302] The device displays the feedback on the screen and informs the user of the recorded content via voice notification, etc. When the user sees the feedback, they can adjust their behavior based on the content. The input is the feedback message, and the output is information provided to change the user's behavior.

[0303] (Application example 2)

[0304] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0305] Conventional systems provided feedback based solely on employee behavioral patterns and performance data, failing to take into account employees' emotional states. This could lead to the accumulation of stress and dissatisfaction, leading to a decline in productivity and motivation. Furthermore, there was a lack of means to properly manage employees' emotions in factories and on production lines, making it difficult to maintain an efficient working environment.

[0306] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for collecting employee behavioral patterns and performance data during work in real time; means for saving the collected data in a database; means for preprocessing the saved data and analyzing it using a performance evaluation model; means for generating individual feedback messages based on the analysis results; means for delivering the generated feedback to the employee's terminal; means for collecting employee facial expression and voice data in real time and analyzing the employee's emotional state using an emotion engine; and means for adjusting an appropriate feedback message taking into account the analysis results of the emotional state. This enables comprehensive performance evaluation and feedback that also takes into account the employee's emotional state.

[0307] A "behavioral pattern" refers to the characteristics and tendencies of a series of actions or behaviors that an employee performs while working.

[0308] "Performance data" refers to data about the efficiency and results of an employee's work, including time to complete a task and the number of errors made.

[0309] A "database" is a computer system for systematically storing and managing collected data.

[0310] "Preprocessing" refers to processes such as data cleaning, normalization, and feature extraction that are performed to convert raw data into an analyzable format.

[0311] "Performance Evaluation Model" means an algorithm or statistical model used to analyze and evaluate employee performance.

[0312] An "individual feedback message" is a message containing specific improvements and advice provided to each employee based on the analysis results.

[0313] "Terminal" refers to devices such as computers and smartphones used by employees.

[0314] An "emotion engine" is software or algorithms that analyze employees' facial expressions and voice data to identify their emotional state.

[0315] "Emotional state" refers to an employee's current emotional state or mood, and can include happiness, sadness, anger, surprise, etc.

[0316] "Analysis results" refers to the results of analysis obtained based on collected data.

[0317] A "feedback message" is a message containing information or advice provided to an employee based on the analysis results.

[0318] "Real-time" refers to data collection, analysis, and feedback occurring immediately, without delay.

[0319] This invention relates to a system that monitors employee performance and emotional states in real time and provides appropriate feedback. Specifically, by collecting employee behavioral patterns and performance data and combining this with the employee's emotional state, it is possible to provide more accurate feedback.

[0320] The hardware required to realize this system includes employee terminals, a server, and cameras and microphones. Terminals include PCs, smartphones, tablets, etc., and are devices used by employees in their daily work. Cameras and microphones are either built into the terminals or connected as external devices. The server is used to collect, store, preprocess, analyze, and generate feedback on data.

[0321] First, when an employee begins work, their device collects behavioral patterns and performance data in real time. Specifically, data such as task completion time and number of errors is automatically recorded. Additionally, the employee's facial expressions and voice are collected using a camera and microphone. The collected data is then sent to a server as appropriate.

[0322] The server then uses a database management system to store the collected data in a database. The stored raw data is then cleaned in a pre-processing stage, where normalization and feature extraction are performed. The pre-processed data is then converted into a format suitable for analysis.

[0323] The server then analyzes the data using machine learning algorithms, taking into account behavioral patterns and performance data, as well as the emotional data analyzed by the emotion engine. Machine learning models, such as random forest classifiers, are used for the analysis.

[0324] Based on the analysis, the server generates personalized feedback messages that reflect both the employee's performance and emotional state, such as "You're performing well, but you seem to be a little stressed. Please take breaks and keep working hard."

[0325] The generated feedback message is sent from the server to the terminal and displayed on the screen, allowing the employee to refer to this feedback and move on to the next task.

[0326] To give a specific example, if User A completes a task in 20 minutes with two errors, the device collects this data in real time, and also uses the camera and microphone to collect User A's facial expressions and voice. The emotion engine generates an analysis result that "User A is feeling a little stressed," and this data is sent to the server. The server stores the data in a database, performs preprocessing, and then analyzes it to predict that "Your performance is good, but you appear to be feeling a little stressed." The server generates feedback that says, "Your performance is good, but you appear to be feeling a little stressed. Please take a break and keep trying," and the device notifies User A of this feedback and displays it on the screen. User A takes the feedback into consideration, takes a break, and begins the next task.

[0327] An example prompt for building this system might be, "Implement a robotic system in Python that monitors the performance and emotional state of factory workers in real time and provides feedback. Use OpenCV for face recognition and TENSORFLOW (registered trademark) for emotion recognition, and generate random performance data. The feedback will be displayed as text on the screen."

[0328] This enables comprehensive performance evaluation and feedback that takes into account employees' emotional state, reducing the burden on managers and efficiently supporting employees in improving their skills and managing their emotions.

[0329] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0330] Step 1:

[0331] The device collects behavioral patterns and performance data in real time while employees are working. Specifically, data such as the time it takes to complete tasks and the number of errors made by employees is recorded. Additionally, a camera and microphone are used to collect facial expressions and voice recordings of employees. This provides behavioral patterns, performance data, and emotional data as input.

[0332] Step 2:

[0333] The device periodically transmits collected behavioral patterns, performance data, and emotional data to the server. Data transmitted from the device to the server includes task completion time, number of errors, facial expression images, and voice data. This allows all necessary data to be collected on the server.

[0334] Step 3:

[0335] The server stores the received data in a database. A database management system is used to systematically organize and store each employee's data. Specifically, the data is inserted into the appropriate tables and stored in a format that is ready for subsequent analysis.

[0336] Step 4:

[0337] The server preprocesses the stored raw data, which includes data cleaning, normalization, feature extraction, for example, imputing missing values ​​and removing outliers, and converting the data into a format suitable for analysis.

[0338] Step 5:

[0339] The server then uses the preprocessed data to perform analysis using a performance evaluation model, which uses machine learning algorithms such as random forest classifiers. Based on the input data (task completion time, number of errors, emotion data, etc.), performance evaluation is performed and the results are obtained.

[0340] Step 6:

[0341] The server uses an emotion engine to analyze the employee's emotional state from facial expressions and voice data. The emotion engine uses machine learning algorithms to identify the emotional state (happiness, sadness, anger, surprise, etc.), which is output as emotion data.

[0342] Step 7:

[0343] The server generates a personalized feedback message based on the performance evaluation and the analysis of the emotional state, such as "Your performance is good, but you seem to be a little stressed. Please take a break and keep working hard."

[0344] Step 8:

[0345] The server delivers the generated feedback message to the terminal, which displays the message on the screen. The employee receives this feedback and checks his / her own performance and emotional state.

[0346] In this way, a system is created that provides comprehensive feedback that takes into account an employee's behavioral patterns, performance data, and emotional state.

[0347] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0348] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0349] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0350] [Second embodiment]

[0351] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0352] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0353] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0354] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0355] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0356] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0357] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0358] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0359] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0360] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0361] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0362] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0363] The present invention provides a system for evaluating employee performance and providing personalized feedback. A specific embodiment of this system will be described below.

[0364] 1. Data Collection

[0365] The device collects data in real time while the user is working. Specifically, it records behavioral patterns and performance data such as task completion time, number of errors, and operation methods used. For example, the moment a user completes a specific task, the device collects detailed data such as "task completion time: 20 minutes" and "number of errors: 2."

[0366] 2. Data storage

[0367] The device sends the collected data at regular intervals to a server, which stores the received data in a database that stores the behavioral patterns and performance data of many users.

[0368] 3. Data Preprocessing

[0369] The server preprocesses the raw data stored in the database, which includes cleaning the data (e.g., imputing missing values ​​and removing outliers), normalizing it, and extracting necessary features. The preprocessed data is then converted into a format suitable for analysis.

[0370] 4. Performance Analysis

[0371] The server analyzes the pre-processed data using a machine learning algorithm. Specifically, it uses a pre-trained performance evaluation model (e.g., a random forest classifier) ​​to predict user performance. This analysis results in a performance evaluation based on each user's behavioral patterns.

[0372] 5. Generate feedback

[0373] The server generates personalized feedback messages based on the analysis, including specific instructions such as "You're performing well, keep it up" or "You're making a lot of errors, so you need to pay attention to a particular skill."

[0374] 6. Feedback Distribution

[0375] The server sends the generated feedback to the terminal, which notifies the user and displays it on the screen, allowing the user to understand their own performance and take specific action to improve.

[0376] Specific examples

[0377] Consider a scenario where User A completes a task in 20 minutes with 2 errors.

[0378] The device collects this data and sends it to the server every 10 minutes.

[0379] The server stores the received data in a database.

[0380] After the data is saved, the server cleans it, removes outliers, and then analyzes it using a machine learning model.

[0381] The analysis results in an evaluation of "good performance."

[0382] The server generates feedback saying, "Performance is good, keep it up."

[0383] Finally, the terminal displays this feedback to User A, who then decides on the next action to take based on the feedback.

[0384] This system automates the evaluation and feedback process, significantly reducing the burden on managers and efficiently supporting employee skill improvement.

[0385] The processing flow will be explained below.

[0386] Step 1:

[0387] The user begins work and performs tasks on their device, such as creating documents and entering data for a new project.

[0388] Step 2:

[0389] The device records user behavior patterns and performance data in real time, including task start and end times, operation types, and the number of errors.

[0390] Step 3:

[0391] The device sends the collected data to the server at regular intervals, for example, uploading the data in batches every 10 minutes.

[0392] Step 4:

[0393] The server stores the received data in a database, allowing the behavioral patterns and performance data of all employees to be centrally managed.

[0394] Step 5:

[0395] The server periodically preprocesses the data stored in the database, which includes data cleaning (e.g., imputing missing values, removing outliers), normalization, feature extraction, etc.

[0396] Step 6:

[0397] The server uses the preprocessed data to analyze it with a machine learning algorithm (e.g., random forest classifier), which then evaluates performance based on user behavior patterns.

[0398] Step 7:

[0399] The server generates personalized feedback messages based on the analysis, such as "Your performance is good, keep it up" or "Your number of errors is high, so you need to pay attention to a particular skill."

[0400] Step 8:

[0401] The server transmits the generated feedback message to the user's terminal.

[0402] Step 9:

[0403] The device notifies the user of the received feedback and displays it on the screen, allowing the user to view the feedback and check their own performance.

[0404] Step 10:

[0405] Users take specific action to improve based on the feedback, such as working more carefully to reduce the number of errors or training in a particular skill.

[0406] This series of steps automates the evaluation and feedback process, providing users with concrete guidelines for improvement.

[0407] Example 1

[0408] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0409] Objectively and efficiently evaluating employee performance and providing personalized feedback is important for helping employees improve their skills and increasing work efficiency. However, traditional evaluation methods often involve subjective judgments and manual evaluations, which require time and effort, making it difficult to provide fast and accurate feedback. Another issue is that it is difficult to generate specific feedback for each employee that identifies areas for improvement.

[0410] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0411] In this invention, the server includes means for collecting user behavioral patterns and performance data during work in real time, means for transmitting the collected data at regular intervals, means for storing the transmitted data in a database, means for preprocessing the stored data, means for analyzing the preprocessed data using a performance evaluation model, means for generating individual feedback messages based on the analysis results, and means for delivering the generated feedback to the user's terminal, thereby enabling automatic and efficient evaluation of employee performance and rapid provision of individual, specific feedback.

[0412] "User" means an individual who uses the System and provides behavioral pattern and performance data.

[0413] A "behavioral pattern" refers to a series of actions or movements that a user makes when operating a system, and includes, for example, clicking operations and key inputs.

[0414] "Performance data" refers to specific numerical data for evaluating a user's work efficiency and work results, such as the time it takes a user to complete a task or the number of errors.

[0415] "Means of collecting data in real time" refers to methods of immediately acquiring and recording data when a user performs an operation, including the use of sensors or software logging functions.

[0416] "Means for sending at regular intervals" refers to means for sending collected data to a server at regular time intervals (e.g., every 10 minutes).

[0417] "Means of storing data in a database" refers to a system or method for systematically organizing and safely storing collected data, such as using a relational database management system (RDBMS).

[0418] "Preprocessing means" refers to a method for removing noise from stored raw data and converting it into a format suitable for analysis, including, for example, imputing missing values ​​and removing outliers.

[0419] "Performance Evaluation Model" means a machine learning algorithm model used to evaluate a user's work efficiency and work results based on collected and pre-processed data, such as a random forest classifier.

[0420] "Means for analyzing" refers to the method of inputting the pre-processed data into a performance evaluation model to analyze and predict user performance.

[0421] A "feedback message" refers to a specific message that is generated based on the analysis results and includes an evaluation of the user and points for improvement.

[0422] "Means for delivering to terminal" refers to a method for transmitting the generated feedback message to a terminal accessible to the user.

[0423] The present invention provides a system for evaluating employee performance and providing personalized feedback. A specific embodiment of this system will be described below.

[0424] This system is mainly composed of three entities: a server, a terminal, and a user.

[0425] Data collection

[0426] The device collects data in real time while the user is working. Specifically, it captures detailed performance data such as the actions the user takes (clicks, typing, movement, etc.), the time it takes to complete a task, and the number of errors. This collection is done using software logging functions and sensors. For example, if a user completes a specific task over a 20-minute period and two errors occur during that time, the device will record this data.

[0427] Sending data

[0428] The device sends the collected data to the server at regular intervals (for example, every 10 minutes). The data is sent using the HTTP protocol and encoded in JSON format.

[0429] Data storage

[0430] The server stores the received data in a database, which stores each user's behavioral patterns and performance data. The database uses a relational database such as MySQL or PostgreSQL.

[0431] Data Preprocessing

[0432] The server preprocesses the raw data stored in the database. Specific preprocessing includes filling in missing values, removing outliers, and normalizing the data. It also extracts the features necessary for analysis. For example, for raw data retrieved from the database, it fills in missing values ​​with the average value, filters outliers, and normalizes each data item to a standard scale.

[0433] Performance Analysis

[0434] The server then uses the pre-processed data to analyze it with a machine learning algorithm, which uses a pre-trained random forest classifier to predict user performance. The pre-processed data is input into the random forest model to calculate a performance evaluation score for each user.

[0435] Generate feedback

[0436] The server generates a personalized feedback message based on the analysis results, including specific instructions such as "Your performance is good. Keep it up," or "Your number of errors is high, so you need to pay attention to a specific skill." This feedback message is dynamically generated using a feedback template based on the analysis results using an embedded sentence generation model.

[0437] Feedback Distribution

[0438] The server sends the generated feedback to the device, which notifies the user and displays it on the screen. The server also sends the generated feedback message in JSON format to the device, which parses the message and displays it in the notification area or on the dashboard.

[0439] Specific examples

[0440] Here is a specific example where User A completed a task in 20 minutes with 2 errors.

[0441] 1. The device records user A's actions and sends the data to the server every 10 minutes.

[0442] 2. The server stores the received data in a database.

[0443] 3. After the data is saved, the server cleans the data and removes outliers, then inputs the preprocessed data into a random forest model for analysis.

[0444] 4. The analysis results show that the performance is good.

[0445] 5. The server generates a feedback message saying, "Performance is good, keep it up."

[0446] 6. Finally, the device displays this feedback to User A, who then decides on the next action.

[0447] This prompt explains the overall system flow and the technical details of each process, automating the employee evaluation and feedback process, reducing the burden on managers and efficiently helping employees improve their skills.

[0448] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0449] Step 1: Data collection

[0450] The device collects data in real time while the user is working. This includes detailed performance data such as actions performed by the user, such as clicking, typing, and moving, as well as the time it took to complete a task and the number of errors. Specifically, when a user clicks the "Save" button, the click event is recorded, and if an error occurs, an error message is also added to the log. This collected data is the input data.

[0451] Input: User action data (clicks, typing, movement), task completion time, number of errors

[0452] Output: Detailed performance data logged

[0453] Step 2: Sending data

[0454] The device sends the collected data to the server at regular intervals (for example, every 10 minutes). The data is encoded in JSON format using the HTTP protocol and sent. Specifically, a timer event occurs every 10 minutes, and the device sends the collected data to the server as an HTTP request.

[0455] Input: Detailed logged performance data

[0456] Output: JSON format data sent to the server

[0457] Step 3: Save your data

[0458] The server stores the received data in a database, which stores each user's behavioral patterns and performance data. Specifically, the server parses the received JSON data and inserts each field into the corresponding table in the database.

[0459] Input: JSON format data sent to the server

[0460] Output: Behavioral patterns and performance data stored in a database

[0461] Step 4: Preprocessing the data

[0462] The server preprocesses the raw data stored in the database. This preprocessing includes filling missing values, removing outliers, normalizing the data, and extracting necessary features. Specifically, for the raw data retrieved from the database, the server fills missing values ​​with the mean value, filters out outliers, and then normalizes each data to a standard scale.

[0463] Input: Behavioral patterns and performance data stored in a database

[0464] Output: Preprocessed data

[0465] Step 5: Analyze performance

[0466] The server then uses the preprocessed data to analyze it with a machine learning algorithm, using a pre-trained random forest classifier to evaluate user performance. Specifically, the preprocessed data is input into the random forest model to calculate a performance evaluation score for each user.

[0467] Input: Preprocessed data

[0468] Output: Performance evaluation score

[0469] Step 6: Generate feedback

[0470] The server generates individual feedback messages based on the analysis results. This evaluates the user's performance based on the analysis results, and dynamically generates feedback templates based on specific improvements and instructions using an embedded sentence generation model. For example, it generates a message such as "Your performance is good. Please keep up the good work."

[0471] Input: Performance evaluation score

[0472] Output: Individual feedback messages

[0473] Step 7: Deliver feedback

[0474] The server sends the generated feedback message to the user's device. The device notifies the user of this feedback and displays it on the screen. Specifically, the server sends the generated feedback message to the device in JSON format, and the device parses the message and displays it in the notification area or on the dashboard.

[0475] Input: Individual feedback message

[0476] Output: Feedback message displayed on the screen

[0477] (Application example 1)

[0478] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0479] Traditionally, employee work efficiency and performance evaluations have often been done manually, which not only lacks objectivity but also requires a great deal of time and effort. Feedback based on evaluation results is also common, making it difficult to provide personalized feedback to individual employees. This makes it difficult to improve employee skills and maintain motivation. This is particularly true in logistics centers and other workplaces, where real-time evaluations and immediate feedback are required, and traditional methods are insufficient.

[0480] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0481] In this invention, the server includes means for collecting employee behavioral patterns and performance data in real time during work, means for storing the collected data in a database, means for preprocessing the stored data and analyzing it using a performance evaluation model, means for generating individual feedback messages based on the analysis results, means for delivering the generated feedback to the employee's terminal, and means for transmitting data to smart glasses that display the feedback messages, thereby enabling real-time work evaluation of employees and immediate individual feedback.

[0482] "Employee" means a person employed to perform a specific job.

[0483] A "behavioral pattern" is a series of actions or movements that an employee tends to take while working.

[0484] "Performance data" refers to data that numerically indicates an employee's work efficiency and results.

[0485] "Real-time" refers to immediacy, with almost no time lag between data collection and processing.

[0486] A "database" is a digital storage device that allows collected data to be stored, managed, and searched in an organized manner.

[0487] "Preprocessing" refers to the process of cleaning data and preparing it into the required format for data analysis.

[0488] A "performance evaluation model" is a mathematical or algorithmic method for analyzing and evaluating employee performance.

[0489] "Analysis" refers to information processing that derives meaning and trends from collected data.

[0490] A "personalized feedback message" is a message that provides specific improvement suggestions or evaluations based on a specific employee's performance.

[0491] "Employee device" means a digital device used by an employee in the course of work.

[0492] "Distribution" refers to the act of electronically transmitting a generated feedback message.

[0493] "Smart glasses" are eyeglass-type digital devices that have a display function and can provide information visually.

[0494] This invention is a system that evaluates the work efficiency and performance of employees in real time at work sites such as logistics centers and provides instant personalized feedback. The system uses smart glasses to collect data while employees are working, analyzes the data on a server, and generates and provides feedback to the smart glasses.

[0495] First, employees wear smart glasses while performing their tasks. The glasses collect real-time data on the employee's behavioral patterns and performance (e.g., work time, number of errors, etc.). This data is collected through an interface within the smart glasses.

[0496] The smart glasses then send the collected data to a server at regular intervals, which stores it in a database that stores the behavioral patterns and performance data of numerous employees.

[0497] The server preprocesses the raw data stored in the database, including cleaning the data (e.g., imputing missing values ​​and removing outliers), normalizing it, and extracting necessary features. The preprocessed data is then converted into a format suitable for analysis.

[0498] The server then uses machine learning algorithms (e.g., random forest classifiers or generative AI models) to analyze the preprocessed data. This results in a performance evaluation based on each employee's behavioral patterns. The performance evaluation model is designed to provide specific evaluations and areas for improvement to improve employees' skills.

[0499] Based on the analysis results, an individual feedback message is generated. The feedback message includes performance evaluation results, areas for improvement, and specific advice for skill improvement. For example, feedback such as "Your picking efficiency is high and you are performing well. Please continue," or "There are many errors, so you need to be careful with certain operations."

[0500] The generated feedback is sent from the server to the smart glasses, which visually display the feedback and allow employees to instantly recognize it, allowing them to understand their own performance in real time and take specific action to improve.

[0501] Specific examples

[0502] Let's say that at a logistics center, Employee A is wearing smart glasses while performing picking work. The smart glasses record the time it takes Employee A to pick up a specific item from a shelf and the number of errors he makes. This data is sent to a server in real time and stored in a database.

[0503] The server preprocesses the data and evaluates Employee A's performance using random forests. The analysis results show that Employee A's picking efficiency is high and the number of errors is low, so feedback is generated saying, "Your picking efficiency is high and your performance is excellent. Please continue." This feedback is sent to the smart glasses so that Employee A can check it while he works.

[0504] Prompt Sentence Examples

[0505] "Data is collected when employees complete specific tasks and sent to a server. The data is cleaned and analyzed using machine learning models. Picking efficiency is evaluated and the results are displayed as feedback on the smart glasses."

[0506] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0507] Step 1:

[0508] The device (smart glasses) collects behavioral patterns and performance data in real time while employees are working. The input is employee movement data, and the output is that this data is temporarily stored in the device. Specific operations include recording employee movements and operations with sensors and collecting data such as the time it takes to complete a task and the number of errors.

[0509] Step 2:

[0510] The device sends the collected data to the server at regular intervals. The input is the behavioral patterns and performance data temporarily stored in the device, and the output is the transmission of this data to the server. Specifically, the data is uploaded from the device to the server via wireless communication.

[0511] Step 3:

[0512] The server stores the received data in a database. The input is the behavioral patterns and performance data received from the terminal, and the output is the accumulation of this data in the database. Specifically, the server periodically writes the data to the database.

[0513] Step 4:

[0514] The server preprocesses the stored data. The input is the raw data stored in the database, and the output is the preprocessed data. Specific operations include cleaning the data (filling in missing values ​​and removing outliers) and normalizing it.

[0515] Step 5:

[0516] The server analyzes the preprocessed data using a performance evaluation model. The input is the preprocessed data, and the output is the employee performance evaluation results. Specifically, the server analyzes the data using a machine learning algorithm (e.g., a random forest classifier or a generative AI model) to evaluate each employee's performance.

[0517] Step 6:

[0518] The server generates individual feedback messages based on the analysis results. The input is the performance evaluation results, and the output is the feedback message. Specifically, the server generates a personalized feedback message for each employee based on the analysis results.

[0519] Step 7:

[0520] The server delivers the generated feedback to the terminal. The input is the feedback message, and the output is the feedback message sent to the terminal. As a specific operation, the feedback message is sent to the smart glasses and displayed.

[0521] Step 8:

[0522] The terminal visually displays the received feedback. The input is the feedback message sent from the server, and the output is the feedback message displayed to the employee. Specifically, the feedback message is displayed as a pop-up on the display of the smart glasses.

[0523] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0524] This invention combines a system that collects and analyzes employee behavioral patterns and performance data in real time to provide individualized feedback with an emotion engine that recognizes the user's emotions. This system enables performance evaluation and feedback that take into account the employee's emotional state.

[0525] 1. Data Collection

[0526] The user starts work and performs tasks on their device, such as creating documents for a new project or entering data. The device records the employee's behavioral patterns and performance data (e.g., task completion time, number of errors, etc.) in real time. In addition, the device is equipped with a camera and microphone, which are used to collect the user's facial expressions and voice.

[0527] 2. Emotion recognition

[0528] The device sends the collected facial and voice data to the emotion engine, which uses machine learning algorithms to analyze the user's emotional state (e.g., joy, sadness, anger, surprise, etc.). The analysis results are also recorded.

[0529] 3. Data storage

[0530] The device periodically transmits collected data (behavioral patterns, performance data, and emotion data) to the server, which stores the data in a database.

[0531] 4. Data Preprocessing

[0532] The server preprocesses the raw data stored in the database, including data cleaning, normalization, and feature extraction, and also converts the sentiment data into a format suitable for analysis.

[0533] 5. Performance Analysis

[0534] The server then uses the preprocessed data to analyze it using a machine learning algorithm (e.g., a random forest classifier). This also takes into account the user's emotional data. For example, high levels of stress or frustration can affect performance evaluation.

[0535] 6. Generate feedback

[0536] The server generates a personalized feedback message based on the analysis results, and adjusts the tone and wording of the feedback based on the emotional data. For example, it might say, "You're performing well, but you seem a little stressed. Please take a break and keep working hard."

[0537] 7. Feedback Distribution

[0538] The server sends the generated feedback message to the device, which notifies the user and displays it on the screen. The user can view this feedback to understand both their own performance and emotional state.

[0539] Specific examples

[0540] Consider a scenario where User A completes a task in 20 minutes with 2 errors.

[0541] The device collects this data, and also uses a camera and microphone to capture the user's facial expressions and voice.

[0542] The emotion engine generates an analysis result that "the user is feeling a little stressed."

[0543] The terminal transmits this data to the server.

[0544] The server stores the data in a database and performs preprocessing.

[0545] The machine learning model predicts that "performance is good, but there is some stress."

[0546] The server generates feedback like, "You're performing well, but you seem a little stressed. Please take a break and keep trying."

[0547] The terminal notifies User A of this feedback and displays it on the screen.

[0548] User A takes the feedback into consideration, takes a break, and works on the next task.

[0549] This system automates the evaluation and feedback process, significantly reducing the burden on managers and efficiently supporting employees in improving their skills and managing their emotions.

[0550] The processing flow will be explained below.

[0551] Step 1:

[0552] The user begins work and performs tasks on their device, such as creating documents and entering data for a new project.

[0553] Step 2:

[0554] The device collects user behavioral patterns and performance data in real time. Specifically, it records the task start time, completion time, operation type, number of errors, etc. For example, if a user completes a task in 20 minutes and two errors occur, the device will record that data.

[0555] Step 3:

[0556] The device also uses its built-in camera and microphone to collect facial and audio data, such as recording changes in facial expression and tone of voice while the user performs a task.

[0557] Step 4:

[0558] The terminal periodically (for example, every 10 minutes) transmits the collected behavioral patterns, performance data, and emotion data to the server.

[0559] Step 5:

[0560] The server stores the received data in a database, including, for example, the user's task completion time, number of errors, facial expression data, and tone of voice.

[0561] Step 6:

[0562] The server preprocesses the stored data, which includes cleaning the data (filling in missing values ​​and removing outliers), normalizing the data, and extracting necessary features, such as emotional features (tension, stress, etc.) from speech data.

[0563] Step 7:

[0564] The server uses the preprocessed data to analyze it with a machine learning algorithm (e.g., a random forest classifier) ​​to predict the user's performance. This analysis takes into account not only behavioral patterns and performance data, but also emotional data.

[0565] Step 8:

[0566] The server generates individual feedback messages based on the analysis results. For example, if the user's performance is good, the server generates a message saying, "Your performance is good. Keep up the good work." In addition, based on the emotional data, the server may generate a message saying, "You seem to be feeling a bit stressed. Take a break and keep working hard."

[0567] Step 9:

[0568] The server sends the generated feedback message to the terminal, for example, the above feedback message to the user's terminal.

[0569] Step 10:

[0570] The device notifies the user of the received feedback message and displays it on the screen, for example, as a pop-up notification or on the dashboard.

[0571] Step 11:

[0572] The user reviews the feedback message and takes specific actions to improve their behavior, such as working more carefully to reduce the number of errors or taking breaks.

[0573] This series of steps automates the evaluation and feedback process, reducing the burden on managers and helping them manage both their employees' performance and their emotional state.

[0574] Example 2

[0575] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0576] While traditional employee evaluation systems collect and evaluate behavioral patterns and performance data, they do not take into account the emotional state of employees, making it difficult to provide appropriate feedback based on the psychological state of employees. Furthermore, by taking emotional factors into account in performance evaluations, it is expected that employee motivation and stress management will be improved, optimizing overall performance.

[0577] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0578] In this invention, the server includes means for collecting employee behavioral patterns and performance data during work in real time, means for transmitting the collected facial and voice data to an emotion recognition engine and analyzing the emotional state, means for storing the collected data and analysis results in a database, means for preprocessing the stored data and analyzing it using a performance evaluation model, means for generating individual feedback messages based on the analysis results, and means for delivering the generated feedback to the employee's terminal. This makes it possible to combine the employee's emotional state with the performance evaluation and provide more accurate and individual feedback.

[0579] "Behavioral patterns" refer to a series of actions and behavioral tendencies exhibited by employees during work hours, and include operation logs and time management data.

[0580] "Performance data" refers to indicators of the results and efficiency of an employee's work, including specific data such as the time it takes to complete a task or the number of errors made.

[0581] "Emotion recognition engine" refers to machine learning models and algorithms that analyze collected facial and voice data to identify an employee's emotional state (e.g., joy, sadness, anger, etc.).

[0582] "Emotional state" refers to the psychological state that employees feel while working, and includes joy, sadness, anger, surprise, stress, etc.

[0583] "Database" refers to a system for efficiently storing and managing collected behavioral patterns, performance data, and emotional state data.

[0584] "Preprocessing" refers to a series of operations performed to convert stored raw data into a form suitable for analysis, including, for example, data cleaning, normalization, and feature extraction.

[0585] A "performance evaluation model" refers to a machine learning algorithm or analytical method used to evaluate an employee's job performance and results.

[0586] "Feedback messages" refer to messages containing advice and evaluations provided to employees based on the analysis results, and indicate specific areas for improvement and opportunities for growth.

[0587] "Devices" refers to devices such as computers and smartphones used by employees to carry out their work, and are used to collect data and display feedback.

[0588] MODE FOR CARRYING OUT THE INVENTION

[0589] This invention combines a system that collects and analyzes employee behavioral patterns and performance data in real time to provide individualized feedback with an emotion engine that recognizes the user's emotions. This system enables performance evaluation and feedback that take into account the employee's emotional state.

[0590] 1. Data Collection

[0591] The user begins work and performs a task on the device, such as creating documents for a new project or entering data. The device records the user's operation log (clicks, keystrokes, etc.) in real time and collects performance data such as the time it took to complete the task and the number of errors. The device is equipped with a built-in camera and microphone, which are used to collect the user's facial expressions and voice. During this process, dedicated software installed on the device runs and saves the data in an appropriate format.

[0592] 2. Emotion recognition

[0593] The device sends the collected facial and voice data to an emotion recognition engine. This emotion engine uses machine learning models to analyze the user's emotional state in real time. Specifically, it uses deep learning to analyze the collected facial data and classify emotions such as "happiness," "sadness," and "anger." Similarly, it analyzes the voice data using voice recognition technology to analyze emotions. The analysis results of the emotion recognition engine are returned to the device as a dataset containing the user's emotional state, which is also recorded.

[0594] 3. Data storage

[0595] The device periodically sends collected data (behavioral patterns, performance data, and emotional data) to the server. The data is sent in encrypted form to ensure security. The server stores the received data in a database. The SQL queries and data format used at this time are in accordance with the system specifications.

[0596] 4. Data Preprocessing

[0597] The server cleans the raw data stored in the database, for example by imputing missing values ​​and detecting and correcting outliers. It also performs data normalization (for example, scaling and encoding categorical data). Specific preprocessing steps are also performed on the emotion data to convert it into a format suitable for analysis (for example, feature extraction and time series data shaping).

[0598] 5. Performance Analysis

[0599] The server then uses the preprocessed data to analyze it with a machine learning algorithm. Specifically, it uses a random forest classifier to evaluate employee performance. Emotional data is also used as a feature to influence the model's predictions. For example, if a user is under high stress, their task efficiency tends to decrease, and this is also reflected in the model.

[0600] 6. Generate feedback

[0601] The server generates personalized feedback messages based on the analysis results, using natural language generation (NLG) technology to select appropriate tones and expressions based on emotional data. For example, a message might be generated that reads, "Your performance is good, but you seem to be feeling a bit stressed. Please take breaks and keep working hard."

[0602] 7. Feedback Distribution

[0603] The server sends the generated feedback message to the device. The device receives it and notifies the user. The device displays the feedback on the screen and notifies the user of the recorded content by voice notification, etc. When the user sees the feedback, they can adjust their behavior based on the content.

[0604] Specific examples

[0605] Consider a scenario in which User A begins a task and completes it in 20 minutes, with two errors. The device collects this data in real time, simultaneously capturing the user's facial expressions and voice using a camera and microphone. The device sends the facial and voice data to an emotion recognition engine, which generates an analysis result indicating that "User A is feeling a little stressed." The device receives this result and sends it to the server. The server stores it in a database and preprocesses the data using Python's Pandas library. Outlier detection algorithms and data cleaning techniques are used. A random forest classifier predicts that User A's performance is "good, but appears to be slightly stressed." NLG technology is used to generate a feedback message stating, "Your performance is good, but you appear to be feeling a little stressed. Please take a break and continue working hard." The server then sends the message to the device, which displays it on the device's screen, informing User A. User A can take this information into consideration and take appropriate measures, such as taking a break.

[0606] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0607] Step 1:

[0608] The user begins work and performs tasks on the device. The user creates documents for a new project, enters data, etc. The device records the user's operation log (clicks, keystrokes, etc.) in real time and collects performance data such as task completion time and number of errors. This data is entered and temporarily stored on the device.

[0609] Step 2:

[0610] The device uses a built-in camera and microphone to collect the user's facial and voice data. This data is also recorded in real time and saved in an appropriate format by dedicated software. This collected facial and voice data is input and temporarily stored within the device.

[0611] Step 3:

[0612] The device sends the collected facial and voice data to an emotion recognition engine, which uses machine learning models to analyze the user's emotional state (happiness, sadness, anger, etc.) in real time. The input for the analysis is facial and voice data, and the output is data indicating the user's emotional state. The analysis results are returned to the device and recorded.

[0613] Step 4:

[0614] The device periodically transmits all collected data (behavioral patterns, performance data, and emotion data) to the server. This uses an encrypted protocol to ensure data security. The transmitted data is the input, and the data received by the server is the output.

[0615] Step 5:

[0616] The server stores the received data in a database, recording the data in an SQL query or appropriate format. The input is the data sent and the output is the data stored in the database.

[0617] Step 6:

[0618] The server preprocesses the raw data stored in the database. It performs missing value imputation, outlier detection and correction, and data normalization (scaling and encoding). The input is raw data, and the output is preprocessed data.

[0619] Step 7:

[0620] The server uses the preprocessed data to perform analysis using a machine learning algorithm and a random forest classifier to evaluate employee performance. The inputs are the preprocessed data and sentiment data, and the output is the performance evaluation results.

[0621] Step 8:

[0622] The server generates individual feedback messages based on the analysis results. It uses natural language generation (NLG) technology to select appropriate tones and expressions based on the emotional data. The input is the performance evaluation results and emotional data, and the output is the feedback message.

[0623] Step 9:

[0624] The server sends the generated feedback message to the terminal, which receives it and notifies the user. The input is the feedback message, and the output is the notification to the user.

[0625] Step 10:

[0626] The device displays the feedback on the screen and informs the user of the recorded content via voice notification, etc. When the user sees the feedback, they can adjust their behavior based on the content. The input is the feedback message, and the output is information provided to change the user's behavior.

[0627] (Application example 2)

[0628] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0629] Conventional systems provided feedback based solely on employee behavioral patterns and performance data, failing to take into account employees' emotional states. This could lead to the accumulation of stress and dissatisfaction, leading to a decline in productivity and motivation. Furthermore, there was a lack of means to properly manage employees' emotions in factories and on production lines, making it difficult to maintain an efficient working environment.

[0630] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for collecting employee behavioral patterns and performance data during work in real time; means for saving the collected data in a database; means for preprocessing the saved data and analyzing it using a performance evaluation model; means for generating individual feedback messages based on the analysis results; means for delivering the generated feedback to the employee's terminal; means for collecting employee facial expression and voice data in real time and analyzing the employee's emotional state using an emotion engine; and means for adjusting an appropriate feedback message taking into account the analysis results of the emotional state. This enables comprehensive performance evaluation and feedback that also takes into account the employee's emotional state.

[0631] A "behavioral pattern" refers to the characteristics and tendencies of a series of actions or behaviors that an employee performs while working.

[0632] "Performance data" refers to data about the efficiency and results of an employee's work, including time to complete a task and the number of errors made.

[0633] A "database" is a computer system for systematically storing and managing collected data.

[0634] "Preprocessing" refers to processes such as data cleaning, normalization, and feature extraction that are performed to convert raw data into an analyzable format.

[0635] "Performance Evaluation Model" means an algorithm or statistical model used to analyze and evaluate employee performance.

[0636] An "individual feedback message" is a message containing specific improvements and advice provided to each employee based on the analysis results.

[0637] "Terminal" refers to devices such as computers and smartphones used by employees.

[0638] An "emotion engine" is software or algorithms that analyze employees' facial expressions and voice data to identify their emotional state.

[0639] "Emotional state" refers to an employee's current emotional state or mood, and can include happiness, sadness, anger, surprise, etc.

[0640] "Analysis results" refers to the results of analysis obtained based on collected data.

[0641] A "feedback message" is a message containing information or advice provided to an employee based on the analysis results.

[0642] "Real-time" refers to data collection, analysis, and feedback occurring immediately, without delay.

[0643] This invention relates to a system that monitors employee performance and emotional states in real time and provides appropriate feedback. Specifically, by collecting employee behavioral patterns and performance data and combining this with the employee's emotional state, it is possible to provide more accurate feedback.

[0644] The hardware required to realize this system includes employee terminals, a server, and cameras and microphones. Terminals include PCs, smartphones, tablets, etc., and are devices used by employees in their daily work. Cameras and microphones are either built into the terminals or connected as external devices. The server is used to collect, store, preprocess, analyze, and generate feedback on data.

[0645] First, when an employee begins work, their device collects behavioral patterns and performance data in real time. Specifically, data such as task completion time and number of errors is automatically recorded. Additionally, the employee's facial expressions and voice are collected using a camera and microphone. The collected data is then sent to a server as appropriate.

[0646] The server then uses a database management system to store the collected data in a database. The stored raw data is then cleaned in a pre-processing stage, where normalization and feature extraction are performed. The pre-processed data is then converted into a format suitable for analysis.

[0647] The server then analyzes the data using machine learning algorithms, taking into account behavioral patterns and performance data, as well as the emotional data analyzed by the emotion engine. Machine learning models, such as random forest classifiers, are used for the analysis.

[0648] Based on the analysis, the server generates personalized feedback messages that reflect both the employee's performance and emotional state, such as "You're performing well, but you seem to be a little stressed. Please take breaks and keep working hard."

[0649] The generated feedback message is sent from the server to the terminal and displayed on the screen, allowing the employee to refer to this feedback and move on to the next task.

[0650] To give a specific example, if User A completes a task in 20 minutes with two errors, the device collects this data in real time, and also uses the camera and microphone to collect User A's facial expressions and voice. The emotion engine generates an analysis result that "User A is feeling a little stressed," and this data is sent to the server. The server stores the data in a database, performs preprocessing, and then analyzes it to predict that "Your performance is good, but you appear to be feeling a little stressed." The server generates feedback that says, "Your performance is good, but you appear to be feeling a little stressed. Please take a break and keep trying," and the device notifies User A of this feedback and displays it on the screen. User A takes the feedback into consideration, takes a break, and begins the next task.

[0651] An example prompt for building this system might be, "Implement a robotic system in Python that monitors the performance and emotional state of factory workers in real time and provides feedback. Use OpenCV for face recognition and TensorFlow for emotion recognition, and generate random performance data. The feedback will be displayed as text on the screen."

[0652] This enables comprehensive performance evaluation and feedback that takes into account employees' emotional state, reducing the burden on managers and efficiently supporting employees in improving their skills and managing their emotions.

[0653] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0654] Step 1:

[0655] The device collects behavioral patterns and performance data in real time while employees are working. Specifically, data such as the time it takes to complete tasks and the number of errors made by employees is recorded. Additionally, a camera and microphone are used to collect facial expressions and voice recordings of employees. This provides behavioral patterns, performance data, and emotional data as input.

[0656] Step 2:

[0657] The device periodically transmits collected behavioral patterns, performance data, and emotional data to the server. Data transmitted from the device to the server includes task completion time, number of errors, facial expression images, and voice data. This allows all necessary data to be collected on the server.

[0658] Step 3:

[0659] The server stores the received data in a database. A database management system is used to systematically organize and store each employee's data. Specifically, the data is inserted into the appropriate tables and stored in a format that is ready for subsequent analysis.

[0660] Step 4:

[0661] The server preprocesses the stored raw data, which includes data cleaning, normalization, feature extraction, for example, imputing missing values ​​and removing outliers, and converting the data into a format suitable for analysis.

[0662] Step 5:

[0663] The server then uses the preprocessed data to perform analysis using a performance evaluation model, which uses machine learning algorithms such as random forest classifiers. Based on the input data (task completion time, number of errors, emotion data, etc.), performance evaluation is performed and the results are obtained.

[0664] Step 6:

[0665] The server uses an emotion engine to analyze the employee's emotional state from facial expressions and voice data. The emotion engine uses machine learning algorithms to identify the emotional state (happiness, sadness, anger, surprise, etc.), which is output as emotion data.

[0666] Step 7:

[0667] The server generates a personalized feedback message based on the performance evaluation and the analysis of the emotional state, such as "Your performance is good, but you seem to be a little stressed. Please take a break and keep working hard."

[0668] Step 8:

[0669] The server delivers the generated feedback message to the terminal, which displays the message on the screen. The employee receives this feedback and checks his / her own performance and emotional state.

[0670] In this way, a system is created that provides comprehensive feedback that takes into account an employee's behavioral patterns, performance data, and emotional state.

[0671] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0672] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0673] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0674] [Third embodiment]

[0675] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0676] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0677] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0678] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0679] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0680] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0681] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0682] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0683] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0684] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0685] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0686] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0687] The present invention provides a system for evaluating employee performance and providing personalized feedback. A specific embodiment of this system will be described below.

[0688] 1. Data Collection

[0689] The device collects data in real time while the user is working. Specifically, it records behavioral patterns and performance data such as task completion time, number of errors, and operation methods used. For example, the moment a user completes a specific task, the device collects detailed data such as "task completion time: 20 minutes" and "number of errors: 2."

[0690] 2. Data storage

[0691] The device sends the collected data at regular intervals to a server, which stores the received data in a database that stores the behavioral patterns and performance data of many users.

[0692] 3. Data Preprocessing

[0693] The server preprocesses the raw data stored in the database, which includes cleaning the data (e.g., imputing missing values ​​and removing outliers), normalizing it, and extracting necessary features. The preprocessed data is then converted into a format suitable for analysis.

[0694] 4. Performance Analysis

[0695] The server analyzes the pre-processed data using a machine learning algorithm. Specifically, it uses a pre-trained performance evaluation model (e.g., a random forest classifier) ​​to predict user performance. This analysis results in a performance evaluation based on each user's behavioral patterns.

[0696] 5. Generate feedback

[0697] The server generates personalized feedback messages based on the analysis, including specific instructions such as "You're performing well, keep it up" or "You're making a lot of errors, so you need to pay attention to a particular skill."

[0698] 6. Feedback Distribution

[0699] The server sends the generated feedback to the terminal, which notifies the user and displays it on the screen, allowing the user to understand their own performance and take specific action to improve.

[0700] Specific examples

[0701] Consider a scenario where User A completes a task in 20 minutes with 2 errors.

[0702] The device collects this data and sends it to the server every 10 minutes.

[0703] The server stores the received data in a database.

[0704] After the data is saved, the server cleans it, removes outliers, and then analyzes it using a machine learning model.

[0705] The analysis results in an evaluation of "good performance."

[0706] The server generates feedback saying, "Performance is good, keep it up."

[0707] Finally, the terminal displays this feedback to User A, who then decides on the next action to take based on the feedback.

[0708] This system automates the evaluation and feedback process, significantly reducing the burden on managers and efficiently supporting employee skill improvement.

[0709] The processing flow will be explained below.

[0710] Step 1:

[0711] The user begins work and performs tasks on their device, such as creating documents and entering data for a new project.

[0712] Step 2:

[0713] The device records user behavior patterns and performance data in real time, including task start and end times, operation types, and the number of errors.

[0714] Step 3:

[0715] The device sends the collected data to the server at regular intervals, for example, uploading the data in batches every 10 minutes.

[0716] Step 4:

[0717] The server stores the received data in a database, allowing the behavioral patterns and performance data of all employees to be centrally managed.

[0718] Step 5:

[0719] The server periodically preprocesses the data stored in the database, which includes data cleaning (e.g., imputing missing values, removing outliers), normalization, feature extraction, etc.

[0720] Step 6:

[0721] The server uses the preprocessed data to analyze it with a machine learning algorithm (e.g., random forest classifier), which then evaluates performance based on user behavior patterns.

[0722] Step 7:

[0723] The server generates personalized feedback messages based on the analysis, such as "Your performance is good, keep it up" or "Your number of errors is high, so you need to pay attention to a particular skill."

[0724] Step 8:

[0725] The server transmits the generated feedback message to the user's terminal.

[0726] Step 9:

[0727] The device notifies the user of the received feedback and displays it on the screen, allowing the user to view the feedback and check their own performance.

[0728] Step 10:

[0729] Users take specific action to improve based on the feedback, such as working more carefully to reduce the number of errors or training in a particular skill.

[0730] This series of steps automates the evaluation and feedback process, providing users with concrete guidelines for improvement.

[0731] Example 1

[0732] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0733] Objectively and efficiently evaluating employee performance and providing personalized feedback is important for helping employees improve their skills and increasing work efficiency. However, traditional evaluation methods often involve subjective judgments and manual evaluations, which require time and effort, making it difficult to provide fast and accurate feedback. Another issue is that it is difficult to generate specific feedback for each employee that identifies areas for improvement.

[0734] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0735] In this invention, the server includes means for collecting user behavioral patterns and performance data during work in real time, means for transmitting the collected data at regular intervals, means for storing the transmitted data in a database, means for preprocessing the stored data, means for analyzing the preprocessed data using a performance evaluation model, means for generating individual feedback messages based on the analysis results, and means for delivering the generated feedback to the user's terminal, thereby enabling automatic and efficient evaluation of employee performance and rapid provision of individual, specific feedback.

[0736] "User" means an individual who uses the System and provides behavioral pattern and performance data.

[0737] A "behavioral pattern" refers to a series of actions or movements that a user makes when operating a system, and includes, for example, clicking operations and key inputs.

[0738] "Performance data" refers to specific numerical data for evaluating a user's work efficiency and work results, such as the time it takes a user to complete a task or the number of errors.

[0739] "Means of collecting data in real time" refers to methods of immediately acquiring and recording data when a user performs an operation, including the use of sensors or software logging functions.

[0740] "Means for sending at regular intervals" refers to means for sending collected data to a server at regular time intervals (e.g., every 10 minutes).

[0741] "Means of storing data in a database" refers to a system or method for systematically organizing and safely storing collected data, such as using a relational database management system (RDBMS).

[0742] "Preprocessing means" refers to a method for removing noise from stored raw data and converting it into a format suitable for analysis, including, for example, imputing missing values ​​and removing outliers.

[0743] "Performance Evaluation Model" means a machine learning algorithm model used to evaluate a user's work efficiency and work results based on collected and pre-processed data, such as a random forest classifier.

[0744] "Means for analyzing" refers to the method of inputting the pre-processed data into a performance evaluation model to analyze and predict user performance.

[0745] A "feedback message" refers to a specific message that is generated based on the analysis results and includes an evaluation of the user and points for improvement.

[0746] "Means for delivering to terminal" refers to a method for transmitting the generated feedback message to a terminal accessible to the user.

[0747] The present invention provides a system for evaluating employee performance and providing personalized feedback. A specific embodiment of this system will be described below.

[0748] This system is mainly composed of three entities: a server, a terminal, and a user.

[0749] Data collection

[0750] The device collects data in real time while the user is working. Specifically, it captures detailed performance data such as the actions the user takes (clicks, typing, movement, etc.), the time it takes to complete a task, and the number of errors. This collection is done using software logging functions and sensors. For example, if a user completes a specific task over a 20-minute period and two errors occur during that time, the device will record this data.

[0751] Sending data

[0752] The device sends the collected data to the server at regular intervals (for example, every 10 minutes). The data is sent using the HTTP protocol and encoded in JSON format.

[0753] Data storage

[0754] The server stores the received data in a database, which stores each user's behavioral patterns and performance data. The database uses a relational database such as MySQL or PostgreSQL.

[0755] Data Preprocessing

[0756] The server preprocesses the raw data stored in the database. Specific preprocessing includes filling in missing values, removing outliers, and normalizing the data. It also extracts the features necessary for analysis. For example, for raw data retrieved from the database, it fills in missing values ​​with the average value, filters outliers, and normalizes each data item to a standard scale.

[0757] Performance Analysis

[0758] The server then uses the pre-processed data to analyze it with a machine learning algorithm, which uses a pre-trained random forest classifier to predict user performance. The pre-processed data is input into the random forest model to calculate a performance evaluation score for each user.

[0759] Generate feedback

[0760] The server generates a personalized feedback message based on the analysis results, including specific instructions such as "Your performance is good. Keep it up," or "Your number of errors is high, so you need to pay attention to a specific skill." This feedback message is dynamically generated using a feedback template based on the analysis results using an embedded sentence generation model.

[0761] Feedback Distribution

[0762] The server sends the generated feedback to the device, which notifies the user and displays it on the screen. The server also sends the generated feedback message in JSON format to the device, which parses the message and displays it in the notification area or on the dashboard.

[0763] Specific examples

[0764] Here is a specific example where User A completed a task in 20 minutes with 2 errors.

[0765] 1. The device records user A's actions and sends the data to the server every 10 minutes.

[0766] 2. The server stores the received data in a database.

[0767] 3. After the data is saved, the server cleans the data and removes outliers, then inputs the preprocessed data into a random forest model for analysis.

[0768] 4. The analysis results show that the performance is good.

[0769] 5. The server generates a feedback message saying, "Performance is good, keep it up."

[0770] 6. Finally, the device displays this feedback to User A, who then decides on the next action.

[0771] This prompt explains the overall system flow and the technical details of each process, automating the employee evaluation and feedback process, reducing the burden on managers and efficiently helping employees improve their skills.

[0772] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0773] Step 1: Data collection

[0774] The device collects data in real time while the user is working. This includes detailed performance data such as actions performed by the user, such as clicking, typing, and moving, as well as the time it took to complete a task and the number of errors. Specifically, when a user clicks the "Save" button, the click event is recorded, and if an error occurs, an error message is also added to the log. This collected data is the input data.

[0775] Input: User action data (clicks, typing, movement), task completion time, number of errors

[0776] Output: Detailed performance data logged

[0777] Step 2: Sending data

[0778] The device sends the collected data to the server at regular intervals (for example, every 10 minutes). The data is encoded in JSON format using the HTTP protocol and sent. Specifically, a timer event occurs every 10 minutes, and the device sends the collected data to the server as an HTTP request.

[0779] Input: Detailed logged performance data

[0780] Output: JSON format data sent to the server

[0781] Step 3: Save your data

[0782] The server stores the received data in a database, which stores each user's behavioral patterns and performance data. Specifically, the server parses the received JSON data and inserts each field into the corresponding table in the database.

[0783] Input: JSON format data sent to the server

[0784] Output: Behavioral patterns and performance data stored in a database

[0785] Step 4: Preprocessing the data

[0786] The server preprocesses the raw data stored in the database. This preprocessing includes filling missing values, removing outliers, normalizing the data, and extracting necessary features. Specifically, for the raw data retrieved from the database, the server fills missing values ​​with the mean value, filters out outliers, and then normalizes each data to a standard scale.

[0787] Input: Behavioral patterns and performance data stored in a database

[0788] Output: Preprocessed data

[0789] Step 5: Analyze performance

[0790] The server then uses the preprocessed data to analyze it with a machine learning algorithm, using a pre-trained random forest classifier to evaluate user performance. Specifically, the preprocessed data is input into the random forest model to calculate a performance evaluation score for each user.

[0791] Input: Preprocessed data

[0792] Output: Performance evaluation score

[0793] Step 6: Generate feedback

[0794] The server generates individual feedback messages based on the analysis results. This evaluates the user's performance based on the analysis results, and dynamically generates feedback templates based on specific improvements and instructions using an embedded sentence generation model. For example, it generates a message such as "Your performance is good. Please keep up the good work."

[0795] Input: Performance evaluation score

[0796] Output: Individual feedback messages

[0797] Step 7: Deliver feedback

[0798] The server sends the generated feedback message to the user's device. The device notifies the user of this feedback and displays it on the screen. Specifically, the server sends the generated feedback message to the device in JSON format, and the device parses the message and displays it in the notification area or on the dashboard.

[0799] Input: Individual feedback message

[0800] Output: Feedback message displayed on the screen

[0801] (Application example 1)

[0802] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0803] Traditionally, employee work efficiency and performance evaluations have often been done manually, which not only lacks objectivity but also requires a great deal of time and effort. Feedback based on evaluation results is also common, making it difficult to provide personalized feedback to individual employees. This makes it difficult to improve employee skills and maintain motivation. This is particularly true in logistics centers and other workplaces, where real-time evaluations and immediate feedback are required, and traditional methods are insufficient.

[0804] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0805] In this invention, the server includes means for collecting employee behavioral patterns and performance data in real time during work, means for storing the collected data in a database, means for preprocessing the stored data and analyzing it using a performance evaluation model, means for generating individual feedback messages based on the analysis results, means for delivering the generated feedback to the employee's terminal, and means for transmitting data to smart glasses that display the feedback messages, thereby enabling real-time work evaluation of employees and immediate individual feedback.

[0806] "Employee" means a person employed to perform a specific job.

[0807] A "behavioral pattern" is a series of actions or movements that an employee tends to take while working.

[0808] "Performance data" refers to data that numerically indicates an employee's work efficiency and results.

[0809] "Real-time" refers to immediacy, with almost no time lag between data collection and processing.

[0810] A "database" is a digital storage device that allows collected data to be stored, managed, and searched in an organized manner.

[0811] "Preprocessing" refers to the process of cleaning data and preparing it into the required format for data analysis.

[0812] A "performance evaluation model" is a mathematical or algorithmic method for analyzing and evaluating employee performance.

[0813] "Analysis" refers to information processing that derives meaning and trends from collected data.

[0814] A "personalized feedback message" is a message that provides specific improvement suggestions or evaluations based on a specific employee's performance.

[0815] "Employee device" means a digital device used by an employee in the course of work.

[0816] "Distribution" refers to the act of electronically transmitting a generated feedback message.

[0817] "Smart glasses" are eyeglass-type digital devices that have a display function and can provide information visually.

[0818] This invention is a system that evaluates the work efficiency and performance of employees in real time at work sites such as logistics centers and provides instant personalized feedback. The system uses smart glasses to collect data while employees are working, analyzes the data on a server, and generates and provides feedback to the smart glasses.

[0819] First, employees wear smart glasses while performing their tasks. The glasses collect real-time data on the employee's behavioral patterns and performance (e.g., work time, number of errors, etc.). This data is collected through an interface within the smart glasses.

[0820] The smart glasses then send the collected data to a server at regular intervals, which stores it in a database that stores the behavioral patterns and performance data of numerous employees.

[0821] The server preprocesses the raw data stored in the database, including cleaning the data (e.g., imputing missing values ​​and removing outliers), normalizing it, and extracting necessary features. The preprocessed data is then converted into a format suitable for analysis.

[0822] The server then uses machine learning algorithms (e.g., random forest classifiers or generative AI models) to analyze the preprocessed data. This results in a performance evaluation based on each employee's behavioral patterns. The performance evaluation model is designed to provide specific evaluations and areas for improvement to improve employees' skills.

[0823] Based on the analysis results, an individual feedback message is generated. The feedback message includes performance evaluation results, areas for improvement, and specific advice for skill improvement. For example, feedback such as "Your picking efficiency is high and you are performing well. Please continue," or "There are many errors, so you need to be careful with certain operations."

[0824] The generated feedback is sent from the server to the smart glasses, which visually display the feedback and allow employees to instantly recognize it, allowing them to understand their own performance in real time and take specific action to improve.

[0825] Specific examples

[0826] Let's say that at a logistics center, Employee A is wearing smart glasses while performing picking work. The smart glasses record the time it takes Employee A to pick up a specific item from a shelf and the number of errors he makes. This data is sent to a server in real time and stored in a database.

[0827] The server preprocesses the data and evaluates Employee A's performance using random forests. The analysis results show that Employee A's picking efficiency is high and the number of errors is low, so feedback is generated saying, "Your picking efficiency is high and your performance is excellent. Please continue." This feedback is sent to the smart glasses so that Employee A can check it while he works.

[0828] Prompt Sentence Examples

[0829] "Data is collected when employees complete specific tasks and sent to a server. The data is cleaned and analyzed using machine learning models. Picking efficiency is evaluated and the results are displayed as feedback on the smart glasses."

[0830] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0831] Step 1:

[0832] The device (smart glasses) collects behavioral patterns and performance data in real time while employees are working. The input is employee movement data, and the output is that this data is temporarily stored in the device. Specific operations include recording employee movements and operations with sensors and collecting data such as the time it takes to complete a task and the number of errors.

[0833] Step 2:

[0834] The device sends the collected data to the server at regular intervals. The input is the behavioral patterns and performance data temporarily stored in the device, and the output is the transmission of this data to the server. Specifically, the data is uploaded from the device to the server via wireless communication.

[0835] Step 3:

[0836] The server stores the received data in a database. The input is the behavioral patterns and performance data received from the terminal, and the output is the accumulation of this data in the database. Specifically, the server periodically writes the data to the database.

[0837] Step 4:

[0838] The server preprocesses the stored data. The input is the raw data stored in the database, and the output is the preprocessed data. Specific operations include cleaning the data (filling in missing values ​​and removing outliers) and normalizing it.

[0839] Step 5:

[0840] The server analyzes the preprocessed data using a performance evaluation model. The input is the preprocessed data, and the output is the employee performance evaluation results. Specifically, the server analyzes the data using a machine learning algorithm (e.g., a random forest classifier or a generative AI model) to evaluate each employee's performance.

[0841] Step 6:

[0842] The server generates individual feedback messages based on the analysis results. The input is the performance evaluation results, and the output is the feedback message. Specifically, the server generates a personalized feedback message for each employee based on the analysis results.

[0843] Step 7:

[0844] The server delivers the generated feedback to the terminal. The input is the feedback message, and the output is the feedback message sent to the terminal. As a specific operation, the feedback message is sent to the smart glasses and displayed.

[0845] Step 8:

[0846] The terminal visually displays the received feedback. The input is the feedback message sent from the server, and the output is the feedback message displayed to the employee. Specifically, the feedback message is displayed as a pop-up on the display of the smart glasses.

[0847] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0848] This invention combines a system that collects and analyzes employee behavioral patterns and performance data in real time to provide individualized feedback with an emotion engine that recognizes the user's emotions. This system enables performance evaluation and feedback that take into account the employee's emotional state.

[0849] 1. Data Collection

[0850] The user starts work and performs tasks on their device, such as creating documents for a new project or entering data. The device records the employee's behavioral patterns and performance data (e.g., task completion time, number of errors, etc.) in real time. In addition, the device is equipped with a camera and microphone, which are used to collect the user's facial expressions and voice.

[0851] 2. Emotion recognition

[0852] The device sends the collected facial and voice data to the emotion engine, which uses machine learning algorithms to analyze the user's emotional state (e.g., joy, sadness, anger, surprise, etc.). The analysis results are also recorded.

[0853] 3. Data storage

[0854] The device periodically transmits collected data (behavioral patterns, performance data, and emotion data) to the server, which stores the data in a database.

[0855] 4. Data Preprocessing

[0856] The server preprocesses the raw data stored in the database, including data cleaning, normalization, and feature extraction, and also converts the sentiment data into a format suitable for analysis.

[0857] 5. Performance Analysis

[0858] The server then uses the preprocessed data to analyze it using a machine learning algorithm (e.g., a random forest classifier). This also takes into account the user's emotional data. For example, high levels of stress or frustration can affect performance evaluation.

[0859] 6. Generate feedback

[0860] The server generates a personalized feedback message based on the analysis results, and adjusts the tone and wording of the feedback based on the emotional data. For example, it might say, "You're performing well, but you seem a little stressed. Please take a break and keep working hard."

[0861] 7. Feedback Distribution

[0862] The server sends the generated feedback message to the device, which notifies the user and displays it on the screen. The user can view this feedback to understand both their own performance and emotional state.

[0863] Specific examples

[0864] Consider a scenario where User A completes a task in 20 minutes with 2 errors.

[0865] The device collects this data, and also uses a camera and microphone to capture the user's facial expressions and voice.

[0866] The emotion engine generates an analysis result that "the user is feeling a little stressed."

[0867] The terminal transmits this data to the server.

[0868] The server stores the data in a database and performs preprocessing.

[0869] The machine learning model predicts that "performance is good, but there is some stress."

[0870] The server generates feedback like, "You're performing well, but you seem a little stressed. Please take a break and keep trying."

[0871] The terminal notifies User A of this feedback and displays it on the screen.

[0872] User A takes the feedback into consideration, takes a break, and works on the next task.

[0873] This system automates the evaluation and feedback process, significantly reducing the burden on managers and efficiently supporting employees in improving their skills and managing their emotions.

[0874] The processing flow will be explained below.

[0875] Step 1:

[0876] The user begins work and performs tasks on their device, such as creating documents and entering data for a new project.

[0877] Step 2:

[0878] The device collects user behavioral patterns and performance data in real time. Specifically, it records the task start time, completion time, operation type, number of errors, etc. For example, if a user completes a task in 20 minutes and two errors occur, the device will record that data.

[0879] Step 3:

[0880] The device also uses its built-in camera and microphone to collect facial and audio data, such as recording changes in facial expression and tone of voice while the user performs a task.

[0881] Step 4:

[0882] The terminal periodically (for example, every 10 minutes) transmits the collected behavioral patterns, performance data, and emotion data to the server.

[0883] Step 5:

[0884] The server stores the received data in a database, including, for example, the user's task completion time, number of errors, facial expression data, and tone of voice.

[0885] Step 6:

[0886] The server preprocesses the stored data, which includes cleaning the data (filling in missing values ​​and removing outliers), normalizing the data, and extracting necessary features, such as emotional features (tension, stress, etc.) from speech data.

[0887] Step 7:

[0888] The server uses the preprocessed data to analyze it with a machine learning algorithm (e.g., a random forest classifier) ​​to predict the user's performance. This analysis takes into account not only behavioral patterns and performance data, but also emotional data.

[0889] Step 8:

[0890] The server generates individual feedback messages based on the analysis results. For example, if the user's performance is good, the server generates a message saying, "Your performance is good. Keep up the good work." In addition, based on the emotional data, the server may generate a message saying, "You seem to be feeling a bit stressed. Take a break and keep working hard."

[0891] Step 9:

[0892] The server sends the generated feedback message to the terminal, for example, the above feedback message to the user's terminal.

[0893] Step 10:

[0894] The device notifies the user of the received feedback message and displays it on the screen, for example, as a pop-up notification or on the dashboard.

[0895] Step 11:

[0896] The user reviews the feedback message and takes specific actions to improve their behavior, such as working more carefully to reduce the number of errors or taking breaks.

[0897] This series of steps automates the evaluation and feedback process, reducing the burden on managers and helping them manage both their employees' performance and their emotional state.

[0898] Example 2

[0899] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0900] While traditional employee evaluation systems collect and evaluate behavioral patterns and performance data, they do not take into account the emotional state of employees, making it difficult to provide appropriate feedback based on the psychological state of employees. Furthermore, by taking emotional factors into account in performance evaluations, it is expected that employee motivation and stress management will be improved, optimizing overall performance.

[0901] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0902] In this invention, the server includes means for collecting employee behavioral patterns and performance data during work in real time, means for transmitting the collected facial and voice data to an emotion recognition engine and analyzing the emotional state, means for storing the collected data and analysis results in a database, means for preprocessing the stored data and analyzing it using a performance evaluation model, means for generating individual feedback messages based on the analysis results, and means for delivering the generated feedback to the employee's terminal. This makes it possible to combine the employee's emotional state with the performance evaluation and provide more accurate and individual feedback.

[0903] "Behavioral patterns" refer to a series of actions and behavioral tendencies exhibited by employees during work hours, and include operation logs and time management data.

[0904] "Performance data" refers to indicators of the results and efficiency of an employee's work, including specific data such as the time it takes to complete a task or the number of errors made.

[0905] "Emotion recognition engine" refers to machine learning models and algorithms that analyze collected facial and voice data to identify an employee's emotional state (e.g., joy, sadness, anger, etc.).

[0906] "Emotional state" refers to the psychological state that employees feel while working, and includes joy, sadness, anger, surprise, stress, etc.

[0907] "Database" refers to a system for efficiently storing and managing collected behavioral patterns, performance data, and emotional state data.

[0908] "Preprocessing" refers to a series of operations performed to convert stored raw data into a form suitable for analysis, including, for example, data cleaning, normalization, and feature extraction.

[0909] A "performance evaluation model" refers to a machine learning algorithm or analytical method used to evaluate an employee's job performance and results.

[0910] "Feedback messages" refer to messages containing advice and evaluations provided to employees based on the analysis results, and indicate specific areas for improvement and opportunities for growth.

[0911] "Devices" refers to devices such as computers and smartphones used by employees to carry out their work, and are used to collect data and display feedback.

[0912] MODE FOR CARRYING OUT THE INVENTION

[0913] This invention combines a system that collects and analyzes employee behavioral patterns and performance data in real time to provide individualized feedback with an emotion engine that recognizes the user's emotions. This system enables performance evaluation and feedback that take into account the employee's emotional state.

[0914] 1. Data Collection

[0915] The user begins work and performs a task on the device, such as creating documents for a new project or entering data. The device records the user's operation log (clicks, keystrokes, etc.) in real time and collects performance data such as the time it took to complete the task and the number of errors. The device is equipped with a built-in camera and microphone, which are used to collect the user's facial expressions and voice. During this process, dedicated software installed on the device runs and saves the data in an appropriate format.

[0916] 2. Emotion recognition

[0917] The device sends the collected facial and voice data to an emotion recognition engine. This emotion engine uses machine learning models to analyze the user's emotional state in real time. Specifically, it uses deep learning to analyze the collected facial data and classify emotions such as "happiness," "sadness," and "anger." Similarly, it analyzes the voice data using voice recognition technology to analyze emotions. The analysis results of the emotion recognition engine are returned to the device as a dataset containing the user's emotional state, which is also recorded.

[0918] 3. Data storage

[0919] The device periodically sends collected data (behavioral patterns, performance data, and emotional data) to the server. The data is sent in encrypted form to ensure security. The server stores the received data in a database. The SQL queries and data format used at this time are in accordance with the system specifications.

[0920] 4. Data Preprocessing

[0921] The server cleans the raw data stored in the database, for example by imputing missing values ​​and detecting and correcting outliers. It also performs data normalization (for example, scaling and encoding categorical data). Specific preprocessing steps are also performed on the emotion data to convert it into a format suitable for analysis (for example, feature extraction and time series data shaping).

[0922] 5. Performance Analysis

[0923] The server then uses the preprocessed data to analyze it with a machine learning algorithm. Specifically, it uses a random forest classifier to evaluate employee performance. Emotional data is also used as a feature to influence the model's predictions. For example, if a user is under high stress, their task efficiency tends to decrease, and this is also reflected in the model.

[0924] 6. Generate feedback

[0925] The server generates personalized feedback messages based on the analysis results, using natural language generation (NLG) technology to select appropriate tones and expressions based on emotional data. For example, a message might be generated that reads, "Your performance is good, but you seem to be feeling a bit stressed. Please take breaks and keep working hard."

[0926] 7. Feedback Distribution

[0927] The server sends the generated feedback message to the device. The device receives it and notifies the user. The device displays the feedback on the screen and notifies the user of the recorded content by voice notification, etc. When the user sees the feedback, they can adjust their behavior based on the content.

[0928] Specific examples

[0929] Consider a scenario in which User A begins a task and completes it in 20 minutes, with two errors. The device collects this data in real time, simultaneously capturing the user's facial expressions and voice using a camera and microphone. The device sends the facial and voice data to an emotion recognition engine, which generates an analysis result indicating that "User A is feeling a little stressed." The device receives this result and sends it to the server. The server stores it in a database and preprocesses the data using Python's Pandas library. Outlier detection algorithms and data cleaning techniques are used. A random forest classifier predicts that User A's performance is "good, but appears to be slightly stressed." NLG technology is used to generate a feedback message stating, "Your performance is good, but you appear to be feeling a little stressed. Please take a break and continue working hard." The server then sends the message to the device, which displays it on the device's screen, informing User A. User A can take this information into consideration and take appropriate measures, such as taking a break.

[0930] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0931] Step 1:

[0932] The user begins work and performs tasks on the device. The user creates documents for a new project, enters data, etc. The device records the user's operation log (clicks, keystrokes, etc.) in real time and collects performance data such as task completion time and number of errors. This data is entered and temporarily stored on the device.

[0933] Step 2:

[0934] The device uses a built-in camera and microphone to collect the user's facial and voice data. This data is also recorded in real time and saved in an appropriate format by dedicated software. This collected facial and voice data is input and temporarily stored within the device.

[0935] Step 3:

[0936] The device sends the collected facial and voice data to an emotion recognition engine, which uses machine learning models to analyze the user's emotional state (happiness, sadness, anger, etc.) in real time. The input for the analysis is facial and voice data, and the output is data indicating the user's emotional state. The analysis results are returned to the device and recorded.

[0937] Step 4:

[0938] The device periodically transmits all collected data (behavioral patterns, performance data, and emotion data) to the server. This uses an encrypted protocol to ensure data security. The transmitted data is the input, and the data received by the server is the output.

[0939] Step 5:

[0940] The server stores the received data in a database, recording the data in an SQL query or appropriate format. The input is the data sent and the output is the data stored in the database.

[0941] Step 6:

[0942] The server preprocesses the raw data stored in the database. It performs missing value imputation, outlier detection and correction, and data normalization (scaling and encoding). The input is raw data, and the output is preprocessed data.

[0943] Step 7:

[0944] The server uses the preprocessed data to perform analysis using a machine learning algorithm and a random forest classifier to evaluate employee performance. The inputs are the preprocessed data and sentiment data, and the output is the performance evaluation results.

[0945] Step 8:

[0946] The server generates individual feedback messages based on the analysis results. It uses natural language generation (NLG) technology to select appropriate tones and expressions based on the emotional data. The input is the performance evaluation results and emotional data, and the output is the feedback message.

[0947] Step 9:

[0948] The server sends the generated feedback message to the terminal, which receives it and notifies the user. The input is the feedback message, and the output is the notification to the user.

[0949] Step 10:

[0950] The device displays the feedback on the screen and informs the user of the recorded content via voice notification, etc. When the user sees the feedback, they can adjust their behavior based on the content. The input is the feedback message, and the output is information provided to change the user's behavior.

[0951] (Application example 2)

[0952] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0953] Conventional systems provided feedback based solely on employee behavioral patterns and performance data, failing to take into account employees' emotional states. This could lead to the accumulation of stress and dissatisfaction, leading to a decline in productivity and motivation. Furthermore, there was a lack of means to properly manage employees' emotions in factories and on production lines, making it difficult to maintain an efficient working environment.

[0954] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for collecting employee behavioral patterns and performance data during work in real time; means for saving the collected data in a database; means for preprocessing the saved data and analyzing it using a performance evaluation model; means for generating individual feedback messages based on the analysis results; means for delivering the generated feedback to the employee's terminal; means for collecting employee facial expression and voice data in real time and analyzing the employee's emotional state using an emotion engine; and means for adjusting an appropriate feedback message taking into account the analysis results of the emotional state. This enables comprehensive performance evaluation and feedback that also takes into account the employee's emotional state.

[0955] A "behavioral pattern" refers to the characteristics and tendencies of a series of actions or behaviors that an employee performs while working.

[0956] "Performance data" refers to data about the efficiency and results of an employee's work, including time to complete a task and the number of errors made.

[0957] A "database" is a computer system for systematically storing and managing collected data.

[0958] "Preprocessing" refers to processes such as data cleaning, normalization, and feature extraction that are performed to convert raw data into an analyzable format.

[0959] "Performance Evaluation Model" means an algorithm or statistical model used to analyze and evaluate employee performance.

[0960] An "individual feedback message" is a message containing specific improvements and advice provided to each employee based on the analysis results.

[0961] "Terminal" refers to devices such as computers and smartphones used by employees.

[0962] An "emotion engine" is software or algorithms that analyze employees' facial expressions and voice data to identify their emotional state.

[0963] "Emotional state" refers to an employee's current emotional state or mood, and can include happiness, sadness, anger, surprise, etc.

[0964] "Analysis results" refers to the results of analysis obtained based on collected data.

[0965] A "feedback message" is a message containing information or advice provided to an employee based on the analysis results.

[0966] "Real-time" refers to data collection, analysis, and feedback occurring immediately, without delay.

[0967] This invention relates to a system that monitors employee performance and emotional states in real time and provides appropriate feedback. Specifically, by collecting employee behavioral patterns and performance data and combining this with the employee's emotional state, it is possible to provide more accurate feedback.

[0968] The hardware required to realize this system includes employee terminals, a server, and cameras and microphones. Terminals include PCs, smartphones, tablets, etc., and are devices used by employees in their daily work. Cameras and microphones are either built into the terminals or connected as external devices. The server is used to collect, store, preprocess, analyze, and generate feedback on data.

[0969] First, when an employee begins work, their device collects behavioral patterns and performance data in real time. Specifically, data such as task completion time and number of errors is automatically recorded. Additionally, the employee's facial expressions and voice are collected using a camera and microphone. The collected data is then sent to a server as appropriate.

[0970] The server then uses a database management system to store the collected data in a database. The stored raw data is then cleaned in a pre-processing stage, where normalization and feature extraction are performed. The pre-processed data is then converted into a format suitable for analysis.

[0971] The server then analyzes the data using machine learning algorithms, taking into account behavioral patterns and performance data, as well as the emotional data analyzed by the emotion engine. Machine learning models, such as random forest classifiers, are used for the analysis.

[0972] Based on the analysis, the server generates personalized feedback messages that reflect both the employee's performance and emotional state, such as "You're performing well, but you seem to be a little stressed. Please take breaks and keep working hard."

[0973] The generated feedback message is sent from the server to the terminal and displayed on the screen, allowing the employee to refer to this feedback and move on to the next task.

[0974] To give a specific example, if User A completes a task in 20 minutes with two errors, the device collects this data in real time, and also uses the camera and microphone to collect User A's facial expressions and voice. The emotion engine generates an analysis result that "User A is feeling a little stressed," and this data is sent to the server. The server stores the data in a database, performs preprocessing, and then analyzes it to predict that "Your performance is good, but you appear to be feeling a little stressed." The server generates feedback that says, "Your performance is good, but you appear to be feeling a little stressed. Please take a break and keep trying," and the device notifies User A of this feedback and displays it on the screen. User A takes the feedback into consideration, takes a break, and begins the next task.

[0975] An example prompt for building this system might be, "Implement a robotic system in Python that monitors the performance and emotional state of factory workers in real time and provides feedback. Use OpenCV for face recognition and TensorFlow for emotion recognition, and generate random performance data. The feedback will be displayed as text on the screen."

[0976] This enables comprehensive performance evaluation and feedback that takes into account employees' emotional state, reducing the burden on managers and efficiently supporting employees in improving their skills and managing their emotions.

[0977] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0978] Step 1:

[0979] The device collects behavioral patterns and performance data in real time while employees are working. Specifically, data such as the time it takes to complete tasks and the number of errors made by employees is recorded. Additionally, a camera and microphone are used to collect facial expressions and voice recordings of employees. This provides behavioral patterns, performance data, and emotional data as input.

[0980] Step 2:

[0981] The device periodically transmits collected behavioral patterns, performance data, and emotional data to the server. Data transmitted from the device to the server includes task completion time, number of errors, facial expression images, and voice data. This allows all necessary data to be collected on the server.

[0982] Step 3:

[0983] The server stores the received data in a database. A database management system is used to systematically organize and store each employee's data. Specifically, the data is inserted into the appropriate tables and stored in a format that is ready for subsequent analysis.

[0984] Step 4:

[0985] The server preprocesses the stored raw data, which includes data cleaning, normalization, feature extraction, for example, imputing missing values ​​and removing outliers, and converting the data into a format suitable for analysis.

[0986] Step 5:

[0987] The server then uses the preprocessed data to perform analysis using a performance evaluation model, which uses machine learning algorithms such as random forest classifiers. Based on the input data (task completion time, number of errors, emotion data, etc.), performance evaluation is performed and the results are obtained.

[0988] Step 6:

[0989] The server uses an emotion engine to analyze the employee's emotional state from facial expressions and voice data. The emotion engine uses machine learning algorithms to identify the emotional state (happiness, sadness, anger, surprise, etc.), which is output as emotion data.

[0990] Step 7:

[0991] The server generates a personalized feedback message based on the performance evaluation and the analysis of the emotional state, such as "Your performance is good, but you seem to be a little stressed. Please take a break and keep working hard."

[0992] Step 8:

[0993] The server delivers the generated feedback message to the terminal, which displays the message on the screen. The employee receives this feedback and checks his / her own performance and emotional state.

[0994] In this way, a system is created that provides comprehensive feedback that takes into account an employee's behavioral patterns, performance data, and emotional state.

[0995] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0996] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0997] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0998] [Fourth embodiment]

[0999] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1000] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1001] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1002] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1003] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1004] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1005] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1006] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1007] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1008] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1009] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1010] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1011] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1012] The present invention provides a system for evaluating employee performance and providing personalized feedback. A specific embodiment of this system will be described below.

[1013] 1. Data Collection

[1014] The device collects data in real time while the user is working. Specifically, it records behavioral patterns and performance data such as task completion time, number of errors, and operation methods used. For example, the moment a user completes a specific task, the device collects detailed data such as "task completion time: 20 minutes" and "number of errors: 2."

[1015] 2. Data storage

[1016] The device sends the collected data at regular intervals to a server, which stores the received data in a database that stores the behavioral patterns and performance data of many users.

[1017] 3. Data Preprocessing

[1018] The server preprocesses the raw data stored in the database, which includes cleaning the data (e.g., imputing missing values ​​and removing outliers), normalizing it, and extracting necessary features. The preprocessed data is then converted into a format suitable for analysis.

[1019] 4. Performance Analysis

[1020] The server analyzes the pre-processed data using a machine learning algorithm. Specifically, it uses a pre-trained performance evaluation model (e.g., a random forest classifier) ​​to predict user performance. This analysis results in a performance evaluation based on each user's behavioral patterns.

[1021] 5. Generate feedback

[1022] The server generates personalized feedback messages based on the analysis, including specific instructions such as "You're performing well, keep it up" or "You're making a lot of errors, so you need to pay attention to a particular skill."

[1023] 6. Feedback Distribution

[1024] The server sends the generated feedback to the terminal, which notifies the user and displays it on the screen, allowing the user to understand their own performance and take specific action to improve.

[1025] Specific examples

[1026] Consider a scenario where User A completes a task in 20 minutes with 2 errors.

[1027] The device collects this data and sends it to the server every 10 minutes.

[1028] The server stores the received data in a database.

[1029] After the data is saved, the server cleans it, removes outliers, and then analyzes it using a machine learning model.

[1030] The analysis results in an evaluation of "good performance."

[1031] The server generates feedback saying, "Performance is good, keep it up."

[1032] Finally, the terminal displays this feedback to User A, who then decides on the next action to take based on the feedback.

[1033] This system automates the evaluation and feedback process, significantly reducing the burden on managers and efficiently supporting employee skill improvement.

[1034] The processing flow will be explained below.

[1035] Step 1:

[1036] The user begins work and performs tasks on their device, such as creating documents and entering data for a new project.

[1037] Step 2:

[1038] The device records user behavior patterns and performance data in real time, including task start and end times, operation types, and the number of errors.

[1039] Step 3:

[1040] The device sends the collected data to the server at regular intervals, for example, uploading the data in batches every 10 minutes.

[1041] Step 4:

[1042] The server stores the received data in a database, allowing the behavioral patterns and performance data of all employees to be centrally managed.

[1043] Step 5:

[1044] The server periodically preprocesses the data stored in the database, which includes data cleaning (e.g., imputing missing values, removing outliers), normalization, feature extraction, etc.

[1045] Step 6:

[1046] The server uses the preprocessed data to analyze it with a machine learning algorithm (e.g., random forest classifier), which then evaluates performance based on user behavior patterns.

[1047] Step 7:

[1048] The server generates personalized feedback messages based on the analysis, such as "Your performance is good, keep it up" or "Your number of errors is high, so you need to pay attention to a particular skill."

[1049] Step 8:

[1050] The server transmits the generated feedback message to the user's terminal.

[1051] Step 9:

[1052] The device notifies the user of the received feedback and displays it on the screen, allowing the user to view the feedback and check their own performance.

[1053] Step 10:

[1054] Users take specific action to improve based on the feedback, such as working more carefully to reduce the number of errors or training in a particular skill.

[1055] This series of steps automates the evaluation and feedback process, providing users with concrete guidelines for improvement.

[1056] Example 1

[1057] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1058] Objectively and efficiently evaluating employee performance and providing personalized feedback is important for helping employees improve their skills and increasing work efficiency. However, traditional evaluation methods often involve subjective judgments and manual evaluations, which require time and effort, making it difficult to provide fast and accurate feedback. Another issue is that it is difficult to generate specific feedback for each employee that identifies areas for improvement.

[1059] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1060] In this invention, the server includes means for collecting user behavioral patterns and performance data during work in real time, means for transmitting the collected data at regular intervals, means for storing the transmitted data in a database, means for preprocessing the stored data, means for analyzing the preprocessed data using a performance evaluation model, means for generating individual feedback messages based on the analysis results, and means for delivering the generated feedback to the user's terminal, thereby enabling automatic and efficient evaluation of employee performance and rapid provision of individual, specific feedback.

[1061] "User" means an individual who uses the System and provides behavioral pattern and performance data.

[1062] A "behavioral pattern" refers to a series of actions or movements that a user makes when operating a system, and includes, for example, clicking operations and key inputs.

[1063] "Performance data" refers to specific numerical data for evaluating a user's work efficiency and work results, such as the time it takes a user to complete a task or the number of errors.

[1064] "Means of collecting data in real time" refers to methods of immediately acquiring and recording data when a user performs an operation, including the use of sensors or software logging functions.

[1065] "Means for sending at regular intervals" refers to means for sending collected data to a server at regular time intervals (e.g., every 10 minutes).

[1066] "Means of storing data in a database" refers to a system or method for systematically organizing and safely storing collected data, such as using a relational database management system (RDBMS).

[1067] "Preprocessing means" refers to a method for removing noise from stored raw data and converting it into a format suitable for analysis, including, for example, imputing missing values ​​and removing outliers.

[1068] "Performance Evaluation Model" means a machine learning algorithm model used to evaluate a user's work efficiency and work results based on collected and pre-processed data, such as a random forest classifier.

[1069] "Means for analyzing" refers to the method of inputting the pre-processed data into a performance evaluation model to analyze and predict user performance.

[1070] A "feedback message" refers to a specific message that is generated based on the analysis results and includes an evaluation of the user and points for improvement.

[1071] "Means for delivering to terminal" refers to a method for transmitting the generated feedback message to a terminal accessible to the user.

[1072] The present invention provides a system for evaluating employee performance and providing personalized feedback. A specific embodiment of this system will be described below.

[1073] This system is mainly composed of three entities: a server, a terminal, and a user.

[1074] Data collection

[1075] The device collects data in real time while the user is working. Specifically, it captures detailed performance data such as the actions the user takes (clicks, typing, movement, etc.), the time it takes to complete a task, and the number of errors. This collection is done using software logging functions and sensors. For example, if a user completes a specific task over a 20-minute period and two errors occur during that time, the device will record this data.

[1076] Sending data

[1077] The device sends the collected data to the server at regular intervals (for example, every 10 minutes). The data is sent using the HTTP protocol and encoded in JSON format.

[1078] Data storage

[1079] The server stores the received data in a database, which stores each user's behavioral patterns and performance data. The database uses a relational database such as MySQL or PostgreSQL.

[1080] Data Preprocessing

[1081] The server preprocesses the raw data stored in the database. Specific preprocessing includes filling in missing values, removing outliers, and normalizing the data. It also extracts the features necessary for analysis. For example, for raw data retrieved from the database, it fills in missing values ​​with the average value, filters outliers, and normalizes each data item to a standard scale.

[1082] Performance Analysis

[1083] The server then uses the pre-processed data to analyze it with a machine learning algorithm, which uses a pre-trained random forest classifier to predict user performance. The pre-processed data is input into the random forest model to calculate a performance evaluation score for each user.

[1084] Generate feedback

[1085] The server generates a personalized feedback message based on the analysis results, including specific instructions such as "Your performance is good. Keep it up," or "Your number of errors is high, so you need to pay attention to a specific skill." This feedback message is dynamically generated using a feedback template based on the analysis results using an embedded sentence generation model.

[1086] Feedback Distribution

[1087] The server sends the generated feedback to the device, which notifies the user and displays it on the screen. The server also sends the generated feedback message in JSON format to the device, which parses the message and displays it in the notification area or on the dashboard.

[1088] Specific examples

[1089] Here is a specific example where User A completed a task in 20 minutes with 2 errors.

[1090] 1. The device records user A's actions and sends the data to the server every 10 minutes.

[1091] 2. The server stores the received data in a database.

[1092] 3. After the data is saved, the server cleans the data and removes outliers, then inputs the preprocessed data into a random forest model for analysis.

[1093] 4. The analysis results show that the performance is good.

[1094] 5. The server generates a feedback message saying, "Performance is good, keep it up."

[1095] 6. Finally, the device displays this feedback to User A, who then decides on the next action.

[1096] This prompt explains the overall system flow and the technical details of each process, automating the employee evaluation and feedback process, reducing the burden on managers and efficiently helping employees improve their skills.

[1097] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1098] Step 1: Data collection

[1099] The device collects data in real time while the user is working. This includes detailed performance data such as actions performed by the user, such as clicking, typing, and moving, as well as the time it took to complete a task and the number of errors. Specifically, when a user clicks the "Save" button, the click event is recorded, and if an error occurs, an error message is also added to the log. This collected data is the input data.

[1100] Input: User action data (clicks, typing, movement), task completion time, number of errors

[1101] Output: Detailed performance data logged

[1102] Step 2: Sending data

[1103] The device sends the collected data to the server at regular intervals (for example, every 10 minutes). The data is encoded in JSON format using the HTTP protocol and sent. Specifically, a timer event occurs every 10 minutes, and the device sends the collected data to the server as an HTTP request.

[1104] Input: Detailed logged performance data

[1105] Output: JSON format data sent to the server

[1106] Step 3: Save your data

[1107] The server stores the received data in a database, which stores each user's behavioral patterns and performance data. Specifically, the server parses the received JSON data and inserts each field into the corresponding table in the database.

[1108] Input: JSON format data sent to the server

[1109] Output: Behavioral patterns and performance data stored in a database

[1110] Step 4: Preprocessing the data

[1111] The server preprocesses the raw data stored in the database. This preprocessing includes filling missing values, removing outliers, normalizing the data, and extracting necessary features. Specifically, for the raw data retrieved from the database, the server fills missing values ​​with the mean value, filters out outliers, and then normalizes each data to a standard scale.

[1112] Input: Behavioral patterns and performance data stored in a database

[1113] Output: Preprocessed data

[1114] Step 5: Analyze performance

[1115] The server then uses the preprocessed data to analyze it with a machine learning algorithm, using a pre-trained random forest classifier to evaluate user performance. Specifically, the preprocessed data is input into the random forest model to calculate a performance evaluation score for each user.

[1116] Input: Preprocessed data

[1117] Output: Performance evaluation score

[1118] Step 6: Generate feedback

[1119] The server generates individual feedback messages based on the analysis results. This evaluates the user's performance based on the analysis results, and dynamically generates feedback templates based on specific improvements and instructions using an embedded sentence generation model. For example, it generates a message such as "Your performance is good. Please keep up the good work."

[1120] Input: Performance evaluation score

[1121] Output: Individual feedback messages

[1122] Step 7: Deliver feedback

[1123] The server sends the generated feedback message to the user's device. The device notifies the user of this feedback and displays it on the screen. Specifically, the server sends the generated feedback message to the device in JSON format, and the device parses the message and displays it in the notification area or on the dashboard.

[1124] Input: Individual feedback message

[1125] Output: Feedback message displayed on the screen

[1126] (Application example 1)

[1127] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1128] Traditionally, employee work efficiency and performance evaluations have often been done manually, which not only lacks objectivity but also requires a great deal of time and effort. Feedback based on evaluation results is also common, making it difficult to provide personalized feedback to individual employees. This makes it difficult to improve employee skills and maintain motivation. This is particularly true in logistics centers and other workplaces, where real-time evaluations and immediate feedback are required, and traditional methods are insufficient.

[1129] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1130] In this invention, the server includes means for collecting employee behavioral patterns and performance data in real time during work, means for storing the collected data in a database, means for preprocessing the stored data and analyzing it using a performance evaluation model, means for generating individual feedback messages based on the analysis results, means for delivering the generated feedback to the employee's terminal, and means for transmitting data to smart glasses that display the feedback messages, thereby enabling real-time work evaluation of employees and immediate individual feedback.

[1131] "Employee" means a person employed to perform a specific job.

[1132] A "behavioral pattern" is a series of actions or movements that an employee tends to take while working.

[1133] "Performance data" refers to data that numerically indicates an employee's work efficiency and results.

[1134] "Real-time" refers to immediacy, with almost no time lag between data collection and processing.

[1135] A "database" is a digital storage device that allows collected data to be stored, managed, and searched in an organized manner.

[1136] "Preprocessing" refers to the process of cleaning data and preparing it into the required format for data analysis.

[1137] A "performance evaluation model" is a mathematical or algorithmic method for analyzing and evaluating employee performance.

[1138] "Analysis" refers to information processing that derives meaning and trends from collected data.

[1139] A "personalized feedback message" is a message that provides specific improvement suggestions or evaluations based on a specific employee's performance.

[1140] "Employee device" means a digital device used by an employee in the course of work.

[1141] "Distribution" refers to the act of electronically transmitting a generated feedback message.

[1142] "Smart glasses" are eyeglass-type digital devices that have a display function and can provide information visually.

[1143] This invention is a system that evaluates the work efficiency and performance of employees in real time at work sites such as logistics centers and provides instant personalized feedback. The system uses smart glasses to collect data while employees are working, analyzes the data on a server, and generates and provides feedback to the smart glasses.

[1144] First, employees wear smart glasses while performing their tasks. The glasses collect real-time data on the employee's behavioral patterns and performance (e.g., work time, number of errors, etc.). This data is collected through an interface within the smart glasses.

[1145] The smart glasses then send the collected data to a server at regular intervals, which stores it in a database that stores the behavioral patterns and performance data of numerous employees.

[1146] The server preprocesses the raw data stored in the database, including cleaning the data (e.g., imputing missing values ​​and removing outliers), normalizing it, and extracting necessary features. The preprocessed data is then converted into a format suitable for analysis.

[1147] The server then uses machine learning algorithms (e.g., random forest classifiers or generative AI models) to analyze the preprocessed data. This results in a performance evaluation based on each employee's behavioral patterns. The performance evaluation model is designed to provide specific evaluations and areas for improvement to improve employees' skills.

[1148] Based on the analysis results, an individual feedback message is generated. The feedback message includes performance evaluation results, areas for improvement, and specific advice for skill improvement. For example, feedback such as "Your picking efficiency is high and you are performing well. Please continue," or "There are many errors, so you need to be careful with certain operations."

[1149] The generated feedback is sent from the server to the smart glasses, which visually display the feedback and allow employees to instantly recognize it, allowing them to understand their own performance in real time and take specific action to improve.

[1150] Specific examples

[1151] Let's say that at a logistics center, Employee A is wearing smart glasses while performing picking work. The smart glasses record the time it takes Employee A to pick up a specific item from a shelf and the number of errors he makes. This data is sent to a server in real time and stored in a database.

[1152] The server preprocesses the data and evaluates Employee A's performance using random forests. The analysis results show that Employee A's picking efficiency is high and the number of errors is low, so feedback is generated saying, "Your picking efficiency is high and your performance is excellent. Please continue." This feedback is sent to the smart glasses so that Employee A can check it while he works.

[1153] Prompt Sentence Examples

[1154] "Data is collected when employees complete specific tasks and sent to a server. The data is cleaned and analyzed using machine learning models. Picking efficiency is evaluated and the results are displayed as feedback on the smart glasses."

[1155] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1156] Step 1:

[1157] The device (smart glasses) collects behavioral patterns and performance data in real time while employees are working. The input is employee movement data, and the output is that this data is temporarily stored in the device. Specific operations include recording employee movements and operations with sensors and collecting data such as the time it takes to complete a task and the number of errors.

[1158] Step 2:

[1159] The device sends the collected data to the server at regular intervals. The input is the behavioral patterns and performance data temporarily stored in the device, and the output is the transmission of this data to the server. Specifically, the data is uploaded from the device to the server via wireless communication.

[1160] Step 3:

[1161] The server stores the received data in a database. The input is the behavioral patterns and performance data received from the terminal, and the output is the accumulation of this data in the database. Specifically, the server periodically writes the data to the database.

[1162] Step 4:

[1163] The server preprocesses the stored data. The input is the raw data stored in the database, and the output is the preprocessed data. Specific operations include cleaning the data (filling in missing values ​​and removing outliers) and normalizing it.

[1164] Step 5:

[1165] The server analyzes the preprocessed data using a performance evaluation model. The input is the preprocessed data, and the output is the employee performance evaluation results. Specifically, the server analyzes the data using a machine learning algorithm (e.g., a random forest classifier or a generative AI model) to evaluate each employee's performance.

[1166] Step 6:

[1167] The server generates individual feedback messages based on the analysis results. The input is the performance evaluation results, and the output is the feedback message. Specifically, the server generates a personalized feedback message for each employee based on the analysis results.

[1168] Step 7:

[1169] The server delivers the generated feedback to the terminal. The input is the feedback message, and the output is the feedback message sent to the terminal. As a specific operation, the feedback message is sent to the smart glasses and displayed.

[1170] Step 8:

[1171] The terminal visually displays the received feedback. The input is the feedback message sent from the server, and the output is the feedback message displayed to the employee. Specifically, the feedback message is displayed as a pop-up on the display of the smart glasses.

[1172] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1173] This invention combines a system that collects and analyzes employee behavioral patterns and performance data in real time to provide individualized feedback with an emotion engine that recognizes the user's emotions. This system enables performance evaluation and feedback that take into account the employee's emotional state.

[1174] 1. Data Collection

[1175] The user starts work and performs tasks on their device, such as creating documents for a new project or entering data. The device records the employee's behavioral patterns and performance data (e.g., task completion time, number of errors, etc.) in real time. In addition, the device is equipped with a camera and microphone, which are used to collect the user's facial expressions and voice.

[1176] 2. Emotion recognition

[1177] The device sends the collected facial and voice data to the emotion engine, which uses machine learning algorithms to analyze the user's emotional state (e.g., joy, sadness, anger, surprise, etc.). The analysis results are also recorded.

[1178] 3. Data storage

[1179] The device periodically transmits collected data (behavioral patterns, performance data, and emotion data) to the server, which stores the data in a database.

[1180] 4. Data Preprocessing

[1181] The server preprocesses the raw data stored in the database, including data cleaning, normalization, and feature extraction, and also converts the sentiment data into a format suitable for analysis.

[1182] 5. Performance Analysis

[1183] The server then uses the preprocessed data to analyze it using a machine learning algorithm (e.g., a random forest classifier). This also takes into account the user's emotional data. For example, high levels of stress or frustration can affect performance evaluation.

[1184] 6. Generate feedback

[1185] The server generates a personalized feedback message based on the analysis results, and adjusts the tone and wording of the feedback based on the emotional data. For example, it might say, "You're performing well, but you seem a little stressed. Please take a break and keep working hard."

[1186] 7. Feedback Distribution

[1187] The server sends the generated feedback message to the device, which notifies the user and displays it on the screen. The user can view this feedback to understand both their own performance and emotional state.

[1188] Specific examples

[1189] Consider a scenario where User A completes a task in 20 minutes with 2 errors.

[1190] The device collects this data, and also uses a camera and microphone to capture the user's facial expressions and voice.

[1191] The emotion engine generates an analysis result that "the user is feeling a little stressed."

[1192] The terminal transmits this data to the server.

[1193] The server stores the data in a database and performs preprocessing.

[1194] The machine learning model predicts that "performance is good, but there is some stress."

[1195] The server generates feedback like, "You're performing well, but you seem a little stressed. Please take a break and keep trying."

[1196] The terminal notifies User A of this feedback and displays it on the screen.

[1197] User A takes the feedback into consideration, takes a break, and works on the next task.

[1198] This system automates the evaluation and feedback process, significantly reducing the burden on managers and efficiently supporting employees in improving their skills and managing their emotions.

[1199] The processing flow will be explained below.

[1200] Step 1:

[1201] The user begins work and performs tasks on their device, such as creating documents and entering data for a new project.

[1202] Step 2:

[1203] The device collects user behavioral patterns and performance data in real time. Specifically, it records the task start time, completion time, operation type, number of errors, etc. For example, if a user completes a task in 20 minutes and two errors occur, the device will record that data.

[1204] Step 3:

[1205] The device also uses its built-in camera and microphone to collect facial and audio data, such as recording changes in facial expression and tone of voice while the user performs a task.

[1206] Step 4:

[1207] The terminal periodically (for example, every 10 minutes) transmits the collected behavioral patterns, performance data, and emotion data to the server.

[1208] Step 5:

[1209] The server stores the received data in a database, including, for example, the user's task completion time, number of errors, facial expression data, and tone of voice.

[1210] Step 6:

[1211] The server preprocesses the stored data, which includes cleaning the data (filling in missing values ​​and removing outliers), normalizing the data, and extracting necessary features, such as emotional features (tension, stress, etc.) from speech data.

[1212] Step 7:

[1213] The server uses the preprocessed data to analyze it with a machine learning algorithm (e.g., a random forest classifier) ​​to predict the user's performance. This analysis takes into account not only behavioral patterns and performance data, but also emotional data.

[1214] Step 8:

[1215] The server generates individual feedback messages based on the analysis results. For example, if the user's performance is good, the server generates a message saying, "Your performance is good. Keep up the good work." In addition, based on the emotional data, the server may generate a message saying, "You seem to be feeling a bit stressed. Take a break and keep working hard."

[1216] Step 9:

[1217] The server sends the generated feedback message to the terminal, for example, the above feedback message to the user's terminal.

[1218] Step 10:

[1219] The device notifies the user of the received feedback message and displays it on the screen, for example, as a pop-up notification or on the dashboard.

[1220] Step 11:

[1221] The user reviews the feedback message and takes specific actions to improve their behavior, such as working more carefully to reduce the number of errors or taking breaks.

[1222] This series of steps automates the evaluation and feedback process, reducing the burden on managers and helping them manage both their employees' performance and their emotional state.

[1223] Example 2

[1224] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1225] While traditional employee evaluation systems collect and evaluate behavioral patterns and performance data, they do not take into account the emotional state of employees, making it difficult to provide appropriate feedback based on the psychological state of employees. Furthermore, by taking emotional factors into account in performance evaluations, it is expected that employee motivation and stress management will be improved, optimizing overall performance.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1227] In this invention, the server includes means for collecting employee behavioral patterns and performance data during work in real time, means for transmitting the collected facial and voice data to an emotion recognition engine and analyzing the emotional state, means for storing the collected data and analysis results in a database, means for preprocessing the stored data and analyzing it using a performance evaluation model, means for generating individual feedback messages based on the analysis results, and means for delivering the generated feedback to the employee's terminal. This makes it possible to combine the employee's emotional state with the performance evaluation and provide more accurate and individual feedback.

[1228] "Behavioral patterns" refer to a series of actions and behavioral tendencies exhibited by employees during work hours, and include operation logs and time management data.

[1229] "Performance data" refers to indicators of the results and efficiency of an employee's work, including specific data such as the time it takes to complete a task or the number of errors made.

[1230] "Emotion recognition engine" refers to machine learning models and algorithms that analyze collected facial and voice data to identify an employee's emotional state (e.g., joy, sadness, anger, etc.).

[1231] "Emotional state" refers to the psychological state that employees feel while working, and includes joy, sadness, anger, surprise, stress, etc.

[1232] "Database" refers to a system for efficiently storing and managing collected behavioral patterns, performance data, and emotional state data.

[1233] "Preprocessing" refers to a series of operations performed to convert stored raw data into a form suitable for analysis, including, for example, data cleaning, normalization, and feature extraction.

[1234] A "performance evaluation model" refers to a machine learning algorithm or analytical method used to evaluate an employee's job performance and results.

[1235] "Feedback messages" refer to messages containing advice and evaluations provided to employees based on the analysis results, and indicate specific areas for improvement and opportunities for growth.

[1236] "Devices" refers to devices such as computers and smartphones used by employees to carry out their work, and are used to collect data and display feedback.

[1237] MODE FOR CARRYING OUT THE INVENTION

[1238] This invention combines a system that collects and analyzes employee behavioral patterns and performance data in real time to provide individualized feedback with an emotion engine that recognizes the user's emotions. This system enables performance evaluation and feedback that take into account the employee's emotional state.

[1239] 1. Data Collection

[1240] The user begins work and performs a task on the device, such as creating documents for a new project or entering data. The device records the user's operation log (clicks, keystrokes, etc.) in real time and collects performance data such as the time it took to complete the task and the number of errors. The device is equipped with a built-in camera and microphone, which are used to collect the user's facial expressions and voice. During this process, dedicated software installed on the device runs and saves the data in an appropriate format.

[1241] 2. Emotion recognition

[1242] The device sends the collected facial and voice data to an emotion recognition engine. This emotion engine uses machine learning models to analyze the user's emotional state in real time. Specifically, it uses deep learning to analyze the collected facial data and classify emotions such as "happiness," "sadness," and "anger." Similarly, it analyzes the voice data using voice recognition technology to analyze emotions. The analysis results of the emotion recognition engine are returned to the device as a dataset containing the user's emotional state, which is also recorded.

[1243] 3. Data storage

[1244] The device periodically sends collected data (behavioral patterns, performance data, and emotional data) to the server. The data is sent in encrypted form to ensure security. The server stores the received data in a database. The SQL queries and data format used at this time are in accordance with the system specifications.

[1245] 4. Data Preprocessing

[1246] The server cleans the raw data stored in the database, for example by imputing missing values ​​and detecting and correcting outliers. It also performs data normalization (for example, scaling and encoding categorical data). Specific preprocessing steps are also performed on the emotion data to convert it into a format suitable for analysis (for example, feature extraction and time series data shaping).

[1247] 5. Performance Analysis

[1248] The server then uses the preprocessed data to analyze it with a machine learning algorithm. Specifically, it uses a random forest classifier to evaluate employee performance. Emotional data is also used as a feature to influence the model's predictions. For example, if a user is under high stress, their task efficiency tends to decrease, and this is also reflected in the model.

[1249] 6. Generate feedback

[1250] The server generates personalized feedback messages based on the analysis results, using natural language generation (NLG) technology to select appropriate tones and expressions based on emotional data. For example, a message might be generated that reads, "Your performance is good, but you seem to be feeling a bit stressed. Please take breaks and keep working hard."

[1251] 7. Feedback Distribution

[1252] The server sends the generated feedback message to the device. The device receives it and notifies the user. The device displays the feedback on the screen and notifies the user of the recorded content by voice notification, etc. When the user sees the feedback, they can adjust their behavior based on the content.

[1253] Specific examples

[1254] Consider a scenario in which User A begins a task and completes it in 20 minutes, with two errors. The device collects this data in real time, simultaneously capturing the user's facial expressions and voice using a camera and microphone. The device sends the facial and voice data to an emotion recognition engine, which generates an analysis result indicating that "User A is feeling a little stressed." The device receives this result and sends it to the server. The server stores it in a database and preprocesses the data using Python's Pandas library. Outlier detection algorithms and data cleaning techniques are used. A random forest classifier predicts that User A's performance is "good, but appears to be slightly stressed." NLG technology is used to generate a feedback message stating, "Your performance is good, but you appear to be feeling a little stressed. Please take a break and continue working hard." The server then sends the message to the device, which displays it on the device's screen, informing User A. User A can take this information into consideration and take appropriate measures, such as taking a break.

[1255] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1256] Step 1:

[1257] The user begins work and performs tasks on the device. The user creates documents for a new project, enters data, etc. The device records the user's operation log (clicks, keystrokes, etc.) in real time and collects performance data such as task completion time and number of errors. This data is entered and temporarily stored on the device.

[1258] Step 2:

[1259] The device uses a built-in camera and microphone to collect the user's facial and voice data. This data is also recorded in real time and saved in an appropriate format by dedicated software. This collected facial and voice data is input and temporarily stored within the device.

[1260] Step 3:

[1261] The device sends the collected facial and voice data to an emotion recognition engine, which uses machine learning models to analyze the user's emotional state (happiness, sadness, anger, etc.) in real time. The input for the analysis is facial and voice data, and the output is data indicating the user's emotional state. The analysis results are returned to the device and recorded.

[1262] Step 4:

[1263] The device periodically transmits all collected data (behavioral patterns, performance data, and emotion data) to the server. This uses an encrypted protocol to ensure data security. The transmitted data is the input, and the data received by the server is the output.

[1264] Step 5:

[1265] The server stores the received data in a database, recording the data in an SQL query or appropriate format. The input is the data sent and the output is the data stored in the database.

[1266] Step 6:

[1267] The server preprocesses the raw data stored in the database. It performs missing value imputation, outlier detection and correction, and data normalization (scaling and encoding). The input is raw data, and the output is preprocessed data.

[1268] Step 7:

[1269] The server uses the preprocessed data to perform analysis using a machine learning algorithm and a random forest classifier to evaluate employee performance. The inputs are the preprocessed data and sentiment data, and the output is the performance evaluation results.

[1270] Step 8:

[1271] The server generates individual feedback messages based on the analysis results. It uses natural language generation (NLG) technology to select appropriate tones and expressions based on the emotional data. The input is the performance evaluation results and emotional data, and the output is the feedback message.

[1272] Step 9:

[1273] The server sends the generated feedback message to the terminal, which receives it and notifies the user. The input is the feedback message, and the output is the notification to the user.

[1274] Step 10:

[1275] The device displays the feedback on the screen and informs the user of the recorded content via voice notification, etc. When the user sees the feedback, they can adjust their behavior based on the content. The input is the feedback message, and the output is information provided to change the user's behavior.

[1276] (Application example 2)

[1277] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1278] Conventional systems provided feedback based solely on employee behavioral patterns and performance data, failing to take into account employees' emotional states. This could lead to the accumulation of stress and dissatisfaction, leading to a decline in productivity and motivation. Furthermore, there was a lack of means to properly manage employees' emotions in factories and on production lines, making it difficult to maintain an efficient working environment.

[1279] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: means for collecting employee behavioral patterns and performance data during work in real time; means for saving the collected data in a database; means for preprocessing the saved data and analyzing it using a performance evaluation model; means for generating individual feedback messages based on the analysis results; means for delivering the generated feedback to the employee's terminal; means for collecting employee facial expression and voice data in real time and analyzing the employee's emotional state using an emotion engine; and means for adjusting an appropriate feedback message taking into account the analysis results of the emotional state. This enables comprehensive performance evaluation and feedback that also takes into account the employee's emotional state.

[1280] A "behavioral pattern" refers to the characteristics and tendencies of a series of actions or behaviors that an employee performs while working.

[1281] "Performance data" refers to data about the efficiency and results of an employee's work, including time to complete a task and the number of errors made.

[1282] A "database" is a computer system for systematically storing and managing collected data.

[1283] "Preprocessing" refers to processes such as data cleaning, normalization, and feature extraction that are performed to convert raw data into an analyzable format.

[1284] "Performance Evaluation Model" means an algorithm or statistical model used to analyze and evaluate employee performance.

[1285] An "individual feedback message" is a message containing specific improvements and advice provided to each employee based on the analysis results.

[1286] "Terminal" refers to devices such as computers and smartphones used by employees.

[1287] An "emotion engine" is software or algorithms that analyze employees' facial expressions and voice data to identify their emotional state.

[1288] "Emotional state" refers to an employee's current emotional state or mood, and can include happiness, sadness, anger, surprise, etc.

[1289] "Analysis results" refers to the results of analysis obtained based on collected data.

[1290] A "feedback message" is a message containing information or advice provided to an employee based on the analysis results.

[1291] "Real-time" refers to data collection, analysis, and feedback occurring immediately, without delay.

[1292] This invention relates to a system that monitors employee performance and emotional states in real time and provides appropriate feedback. Specifically, by collecting employee behavioral patterns and performance data and combining this with the employee's emotional state, it is possible to provide more accurate feedback.

[1293] The hardware required to realize this system includes employee terminals, a server, and cameras and microphones. Terminals include PCs, smartphones, tablets, etc., and are devices used by employees in their daily work. Cameras and microphones are either built into the terminals or connected as external devices. The server is used to collect, store, preprocess, analyze, and generate feedback on data.

[1294] First, when an employee begins work, their device collects behavioral patterns and performance data in real time. Specifically, data such as task completion time and number of errors is automatically recorded. Additionally, the employee's facial expressions and voice are collected using a camera and microphone. The collected data is then sent to a server as appropriate.

[1295] The server then uses a database management system to store the collected data in a database. The stored raw data is then cleaned in a pre-processing stage, where normalization and feature extraction are performed. The pre-processed data is then converted into a format suitable for analysis.

[1296] The server then analyzes the data using machine learning algorithms, taking into account behavioral patterns and performance data, as well as the emotional data analyzed by the emotion engine. Machine learning models, such as random forest classifiers, are used for the analysis.

[1297] Based on the analysis, the server generates personalized feedback messages that reflect both the employee's performance and emotional state, such as "You're performing well, but you seem to be a little stressed. Please take breaks and keep working hard."

[1298] The generated feedback message is sent from the server to the terminal and displayed on the screen, allowing the employee to refer to this feedback and move on to the next task.

[1299] To give a specific example, if User A completes a task in 20 minutes with two errors, the device collects this data in real time, and also uses the camera and microphone to collect User A's facial expressions and voice. The emotion engine generates an analysis result that "User A is feeling a little stressed," and this data is sent to the server. The server stores the data in a database, performs preprocessing, and then analyzes it to predict that "Your performance is good, but you appear to be feeling a little stressed." The server generates feedback that says, "Your performance is good, but you appear to be feeling a little stressed. Please take a break and keep trying," and the device notifies User A of this feedback and displays it on the screen. User A takes the feedback into consideration, takes a break, and begins the next task.

[1300] An example prompt for building this system might be, "Implement a robotic system in Python that monitors the performance and emotional state of factory workers in real time and provides feedback. Use OpenCV for face recognition and TensorFlow for emotion recognition, and generate random performance data. The feedback will be displayed as text on the screen."

[1301] This enables comprehensive performance evaluation and feedback that takes into account employees' emotional state, reducing the burden on managers and efficiently supporting employees in improving their skills and managing their emotions.

[1302] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1303] Step 1:

[1304] The device collects behavioral patterns and performance data in real time while employees are working. Specifically, data such as the time it takes to complete tasks and the number of errors made by employees is recorded. Additionally, a camera and microphone are used to collect facial expressions and voice recordings of employees. This provides behavioral patterns, performance data, and emotional data as input.

[1305] Step 2:

[1306] The device periodically transmits collected behavioral patterns, performance data, and emotional data to the server. Data transmitted from the device to the server includes task completion time, number of errors, facial expression images, and voice data. This allows all necessary data to be collected on the server.

[1307] Step 3:

[1308] The server stores the received data in a database. A database management system is used to systematically organize and store each employee's data. Specifically, the data is inserted into the appropriate tables and stored in a format that is ready for subsequent analysis.

[1309] Step 4:

[1310] The server preprocesses the stored raw data, which includes data cleaning, normalization, feature extraction, for example, imputing missing values ​​and removing outliers, and converting the data into a format suitable for analysis.

[1311] Step 5:

[1312] The server then uses the preprocessed data to perform analysis using a performance evaluation model, which uses machine learning algorithms such as random forest classifiers. Based on the input data (task completion time, number of errors, emotion data, etc.), performance evaluation is performed and the results are obtained.

[1313] Step 6:

[1314] The server uses an emotion engine to analyze the employee's emotional state from facial expressions and voice data. The emotion engine uses machine learning algorithms to identify the emotional state (happiness, sadness, anger, surprise, etc.), which is output as emotion data.

[1315] Step 7:

[1316] The server generates a personalized feedback message based on the performance evaluation and the analysis of the emotional state, such as "Your performance is good, but you seem to be a little stressed. Please take a break and keep working hard."

[1317] Step 8:

[1318] The server delivers the generated feedback message to the terminal, which displays the message on the screen. The employee receives this feedback and checks his / her own performance and emotional state.

[1319] In this way, a system is created that provides comprehensive feedback that takes into account an employee's behavioral patterns, performance data, and emotional state.

[1320] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1321] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1322] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1323] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1324] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1325] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1326] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1327] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1328] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1329] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1330] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1331] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1332] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1333] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1334] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1335] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1336] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1337] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1338] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1339] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1340] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1341] The following is further disclosed regarding the above embodiment.

[1342] (Claim 1)

[1343] A means of collecting employee work behavioral patterns and performance data in real time;

[1344] a means for storing the collected data in a database;

[1345] means for preprocessing the stored data and analyzing it using a performance evaluation model;

[1346] means for generating a personalized feedback message based on the analysis results;

[1347] a means for delivering the generated feedback to an employee's device;

[1348] A system including:

[1349] (Claim 2)

[1350] 10. The system of claim 1, wherein the personalized feedback message identifies specific areas for improvement and growth opportunities to enhance the employee's skills.

[1351] (Claim 3)

[1352] The system of claim 1 , wherein the performance evaluation model uses a machine learning algorithm.

[1353] "Example 1"

[1354] (Claim 1)

[1355] A means for collecting user behavioral patterns and performance data in real time while working;

[1356] means for transmitting the collected data at regular intervals;

[1357] a means for storing the transmitted data in a database;

[1358] means for preprocessing the stored data;

[1359] means for analyzing the preprocessed data using a performance evaluation model;

[1360] means for generating a personalized feedback message based on the analysis results;

[1361] means for delivering the generated feedback to a user terminal;

[1362] A system including:

[1363] (Claim 2)

[1364] 10. The system of claim 1, wherein the personalized feedback message identifies specific improvement and growth opportunities for the user to enhance their skills.

[1365] (Claim 3)

[1366] The system of claim 1 , wherein the performance evaluation model uses a machine learning algorithm.

[1367] "Application Example 1"

[1368] (Claim 1)

[1369] A means of collecting employee work behavioral patterns and performance data in real time;

[1370] a means for storing the collected data in a database;

[1371] means for preprocessing the stored data and analyzing it using a performance evaluation model;

[1372] means for generating a personalized feedback message based on the analysis results;

[1373] a means for delivering the generated feedback to an employee's device;

[1374] means for transmitting data to smart glasses that display feedback messages;

[1375] A system including:

[1376] (Claim 2)

[1377] 10. The system of claim 1, wherein the personalized feedback message identifies specific areas for improvement and growth opportunities to enhance the employee's skills.

[1378] (Claim 3)

[1379] The system of claim 1 , wherein the performance evaluation model uses a machine learning algorithm.

[1380] "Example 2: Combining Emotion Engines"

[1381] (Claim 1)

[1382] A means of collecting employee work behavioral patterns and performance data in real time;

[1383] means for transmitting the collected facial expression and voice data to an emotion recognition engine for analyzing the emotional state;

[1384] A means for storing the collected data and analysis results in a database;

[1385] means for preprocessing the stored data and analyzing it using a performance evaluation model;

[1386] means for generating a personalized feedback message based on the analysis results;

[1387] a means for delivering the generated feedback to an employee's device;

[1388] A system including:

[1389] (Claim 2)

[1390] The system of claim 1 , wherein the personalized feedback message adjusts tone and presentation based on the employee's emotional state.

[1391] (Claim 3)

[1392] The system of claim 1 , wherein the performance evaluation model uses a machine learning algorithm.

[1393] "Application example 2 when combining emotion engines"

[1394] (Claim 1)

[1395] A means of collecting employee work behavioral patterns and performance data in real time;

[1396] a means for storing the collected data in a database;

[1397] means for preprocessing the stored data and analyzing it using a performance evaluation model;

[1398] means for generating a personalized feedback message based on the analysis results;

[1399] a means for delivering the generated feedback to an employee's device;

[1400] A means of collecting facial expressions and voice data of employees in real time and analyzing their emotional state using an emotion engine;

[1401] means for tailoring appropriate feedback messages taking into account the analysis of the emotional state;

[1402] A system including:

[1403] (Claim 2)

[1404] 10. The system of claim 1, wherein the personalized feedback message identifies specific areas for improvement and growth opportunities to enhance the employee's skills.

[1405] (Claim 3)

[1406] The system of claim 1 , wherein the performance evaluation model uses a machine learning algorithm. [Explanation of symbols]

[1407] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of collecting employee work behavioral patterns and performance data in real time; a means for storing the collected data in a database; means for preprocessing the stored data and analyzing it using a performance evaluation model; means for generating a personalized feedback message based on the analysis results; a means for delivering the generated feedback to an employee's device; A system including:

2. The system of claim 1 , wherein the personalized feedback message identifies specific improvement and growth opportunities for the employee to enhance their skills.

3. The system of claim 1 , wherein the performance evaluation model uses a machine learning algorithm.

Images